Suraj
uploaded a video where he wrote a script to be notified when a specified youtube channel uploads a new video. In the video, he takes advantage of youtube's api in order to scrape for the id of the latest video upload of a channel. He uses several pipes to accomplish this task, and while there's nothing wrong with the one-liner that was used, I just wanted to show off grep's -P option, which is used to enable features from Perl's regex engine.
Just like in the video, we need to start by getting the channel's id. We can very easily scrape for the channel_id of a youtube channel with the following:
$ curl -s 'https://www.youtube.com/@{user-id}' | grep -Po "(?<=channelId\":\")[^\"]+(?=\",\"title\":\")" UCXXXXXXXXXXXXXXXXXXXXXX This is where grep's -P command comes in handy. Perl's regex engine has something called "lookarounds". These are
zero-width assertions that allow for creating
custom anchors in order to narrow down the conditions for a pattern match. If you just read that and you are already familiar with regex, yet you have no idea what I'm talking, allow me to explain further.
If you know a bit of regex, then you are likely familiar with the anchors ^ and $. These too are zero-width assertions used to narrow down the conditions for a pattern match. For something like ^foobar$, the way the regex engine works is it finds the string being processed, in this case foobar, and then "looks around" the string to see if it can
assert that it is in fact at the beginning of a line (^) AND at the end of the line ($) in order to successfully match the pattern. ^ and $ are zero-width as they do not match any characters, they simply look around from the string to assert what comes before and after.
Let's take away the lookarounds from the original grep and see exactly what we're matching:
$ curl -s 'https://www.youtube.com/@{user-id}' | grep -Po "channelId\":\"[^\"]+\",\"title\":\"" channelId":"UCXXXXXXXXXXXXXXXXXXXXXX","title":" We're looking for a pattern that matches the literal string channelId":", followed by [^\"]+ (one or more NOT quotation mark characters, which will capture the channel id), ending with the literal string ","title":". We have to use the regex for one or more NOT quotation marks characters instead of something more straightforward like .+ because the pattern we're matching just so happens to be on a very long single line, and the specificity will constrain regex's greedy nature from matching something further along that line that we don't mean for it to match, and it will. In this instance, .{24} would also suffice as that is saying to match 24 of any character, the exact length of youtube channel ids.
We only want the channel id string though. You can see that it is "anchored" so to speak by the preceding string channelId":" followed by ","title":". Time to make use of a feature from Perl's regex engine to lookaround.
Back to the original grep command:
grep -Po "(?<=channelId\":\")[^\"]+(?=\",\"title\":\")"
Note that the parenthetical groups are back, and not only that, they also contain some very specific constructs: ?<= and ?=. This is the syntax for
Positive lookbehind and
Positive lookahead respectively. Positive because the pattern match within the assertion must succeed(be positively identified). Negative lookahead and lookbehind exist as well(good for when you want to match only if the assertion fails). These are what create custom anchors and turn the strings channelId":" and ","title":" into zero-width assertions. Think back to ^foobar$. When foobar is matched, the regex engine will
look behind foobar to see if it's anchored to the start of a line, and
look ahead of foobar to see if it's anchored to the end.
With the grep command above, we are processing the string [^\"]+, and we are looking behind (?<=) the string to see if it is anchored to channelId":" and we are looking ahead (?=) of it to see if it's anchored to ","title":". If the assertions are satisfied, the string we are looking for and only that string will be printed. The anchors have served their purpose as they only look around the pattern and do not match any characters themselves.
Hopefully with that introduction to perl's lookaround constructs you have more of an understanding of how they work if you didn't before.
Let's press forward with the final grep command for extracting video-ids from the xml output of the api response:
$ curl -s 'https://www.youtube.com/feeds/videos.xml?channel_id=UCXXXXXXXXXXXXXXXXXXXXXX' | grep -oP "(?<=)[^<]+(?=)" xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx
This one is more straightforward since the xml is nice and organized. We create custom anchors to lookaround for. From the string [^<]+ , which will be the video id (though .{11} and now .+ would work too), we will lookbehind (?<=) for the literal string
and lookahead (?=) for the literal string . If the assertions are satisified, the 15 video ids will be returned.
If you just want the latest video:
$ curl -s 'https://www.youtube.com/feeds/videos.xml?channel_id=UCXXXXXXXXXXXXXXXXXXXXXX' | grep -m1 -oP "(?<=)[^<]+(?=)" xxxxxxxxxxx
And voila, we got just the output we wanted using grep only.
I just learned about this so I am satisfied with just writing this up as a way to reinforce my own understanding on something I know I will return to frequently, but I'm also happy to share this knowledge with some dank linux users in hopes that they too will benefit from it.
I am not exactly a grep master and I know nothing about perl beyond what I just shared right now. I wrote this up using the following for reference:
GNU GREP and RIPGREP (learnbyexample) - Perl Compatible Regular Expressions Perl Monks - Using Look-ahead and Look-behind -- grep E-mails grep -EiEio '\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b' emails.txt -- или так найти все e-mail grep -o '[[:alnum:]+\.\_\-]*@[[:alnum:]+\.\_\-]*' emails.txt -- выбрать все IP адреса grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" /valog/auth.log -- исключить шаблон grep -v string-to-exclude filename -- исключить процессы пользователя root ps -efH | grep -v '^root' -- без процессов root и zabbix ps -efH | grep -v '^root' | grep -v '^zabbix' -- убрать комментарии и пустые строки grep -v '^#|^$' example.conf
submitted by ستحتاج إلى الانتقال إلى صفحة إصدارات Github لـ code-server واختيار أحدث إصدار من Linux (سيحتوي اسم الملف على الكلمة "linux"). كان الإصدار الأخير عند كتابة هذا التقرير هو 2.1692. نزّله باستخدام في ملفات Linux (بالفعل جميع unicies) يتم إنشاؤها عند فتحها وحذفها عندما لا شيء يحمل إشارة إليها. في هذه الحالة ، البرنامج الذي فتحه والدليل فتحه 'في' مراجع المراجع إلى الملف. عندما يريد برنامج cp ... و تمرير اسم الملف كمعامل (طالما ان الملف موجود في الدليل الحالي) . طباعة المخرجات علي الشاشة. يستخدم الامر echo لطباعة سلاسل الحروف و قيم المتغيرات المختلفة علي شاشة الغلاف كالتالي: echo str. echo "Welcome to Bash” من الممكن استخدام او ... و تمرير اسم الملف كمعامل (طالما ان الملف موجود في الدليل الحالي) . طباعة المخرجات علي الشاشة. يستخدم الامر echo لطباعة سلاسل الحروف و قيم المتغيرات المختلفة علي شاشة الغلاف كالتالي: echo str. echo "Welcome to Bash” من الممكن استخدام او ... linux - شرح - what are grep patterns called? كيفية منع نتائج مطابقة الملف الثنائي في grep (2) عند استخدام grep في linux ، فإن النتيجة تحتوي دائمًا على الكثير من "تطابقات ملف XXX الثنائية" ، والتي لا يهمني. كيفية قمع هذا الجزء من النتائج ، أو كيفية ...
