openresty / openresty/lua-nginx-module
ngx.re.gsub not working as expected
Open
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 11.8k
- Forks
- 2.1k
- Avg merge
- 6h 1m
- Merged PRs (30d)
- 6
Description
OpenResty version: openresty-1.15.8.3
Mac OS Catalina 10.15.3
I have created a PCRE regex for fixing image src values in a regex.
You can see the regex working here: https://regexr.com/51bj1
In the correct implementation of PCRE regex you can see that this expression takes a URL parameter from the <a href= segment, and replaces the nested <img src= value with the capture from the link.
When I run the code below, using the same HTML example, and ngx.re.gsub, the result is that it does not replace all 4 iterations of the link with nested images, but it takes the <a href= capture from the first iteration, ignores the rest of the matches, and only applies the 1st capture to the last match.
local function fix_images(htmlstr)
local newstr, n, err = ngx.re.gsub(htmlstr
, "(?<=<a class=\"BVG0Nb\" href=\"\\/imgres\\?imgurl=)([\\%\\,\\--\\/0-9\\:\\;\\=\\?\\@A-Z_a-z]+)(&.*\"><div><img class=\"WddBJd\" .* )src=(\"([\\%\\,\\--\\/0-9\\:\\;\\=\\@A-Z_a-z]+)\")(?= id.*><\\/div><\\/a>)"
, "$1$2src=\"$1\""
, "U"
)
if newstr then
return newstr
else
ngx.log(ngx.ERR, "filter fix_images: ", err)
return htmlstr
end
end
local htmlstr = "<div class=\"Xdlr0d\"><div class=\"idg8be\"><a class=\"BVG0Nb\" href=\"/imgres?imgurl=http://www.gstatic.com/tv/thumb/tvbanners/7892603/p7892603_b_v8_ad.jpg&imgrefurl=http://google.com/search?tbm=isch&q=Friends&h=1440&w=960&tbnid=7i2_XywEZ60b2M:&q=Friends&tbnh=186&tbnw=124&usg=AI4_-kS-26wWdmwxNKHnvDZR4tXHgB8oRw&vet=1&docid=XJI53M_S0qN5PM&itg=1&hl=en&sa=X&ved=2ahUKEwiNtdDi6MDoAhUJt54KHfkfACoQ9QF6BAgUEAQ\"><div><img class=\"WddBJd\" style=\"max-width:none;height:128px;max-height:128px\" alt=\"www.gstatic.com/tv/thumb/tvbanners/7892603/p789260...\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==\" id=\"dimg_29\" data-deferred=\"1\" data-iml=\"1585523892832\"></div></a><a class=\"BVG0Nb\" href=\"/imgres?imgurl=https://m.media-amazon.com/images/M/MV5BNDVkYjU0MzctMWRmZi00NTkxLTgwZWEtOWVhYjZlYjllYmU4XkEyXkFqcGdeQXVyNTA4NzY1MzY@._V1_UY1200_CR93,0,630,1200_AL_.jpg&imgrefurl=https://www.imdb.com/title/tt0108778/&h=1200&w=630&tbnid=_oJOKB4azrRhBM:&q=Friends&tbnh=98&tbnw=51&usg=AI4_-kTTSv2chNcoSF0f9JBQGg3jMm2i1Q&vet=1&docid=OXs4KB9tMx4sIM&hl=en&sa=X&ved=2ahUKEwiNtdDi6MDoAhUJt54KHfkfACoQ9QF6BAgUEAY\"><div><img class=\"WddBJd\" style=\"max-width:none;height:128px;max-height:128px\" alt=\"m.media-amazon.com/images/M/MV5BNDVkYjU0MzctMWRmZi...\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==\" id=\"dimg_31\" data-deferred=\"1\" data-iml=\"1585523892832\"></div></a><a class=\"BVG0Nb\" href=\"/imgres?imgurl=https://static01.nyt.com/images/2020/02/21/multimedia/21xp-friends/21xp-friends-mediumSquareAt3X.jpg&imgrefurl=https://www.nytimes.com/2020/02/21/arts/hbo-max-friends-reunion.html&h=1799&w=1800&tbnid=lbyqPRixFQczNM:&q=Friends&tbnh=98&tbnw=98&usg=AI4_-kSg-e73Y4terP_KJDl-ke0ev_NTgg&vet=1&docid=dlGTY7jkGCtE_M&hl=en&sa=X&ved=2ahUKEwiNtdDi6MDoAhUJt54KHfkfACoQ9QF6BAgUEAg\"><div><img class=\"WddBJd\" style=\"max-width:none;height:128px;max-height:128px\" alt=\"static01.nyt.com/images/2020/02/21/multimedia/21xp...\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==\" id=\"dimg_33\" data-deferred=\"1\" data-iml=\"1585523892832\"></div></a><a class=\"BVG0Nb\" href=\"/imgres?imgurl=https://pmcvariety.files.wordpress.com/2020/02/friends.jpg?w=1000&imgrefurl=https://variety.com/2020/tv/news/friends-cast-reunion-hbo-max-1203510743/&h=563&w=1000&tbnid=7WtlGdFDXooGuM:&q=Friends&tbnh=98&tbnw=174&usg=AI4_-kQFuG5qdui9QRsqjY7Kq4unEosqSA&vet=1&docid=UtMcQG_8oL6A0M&hl=en&sa=X&ved=2ahUKEwiNtdDi6MDoAhUJt54KHfkfACoQ9QF6BAgUEAo\"><div><img class=\"WddBJd\" style=\"max-width:none;height:128px;max-height:128px\" alt=\"pmcvariety.files.wordpress.com/2020/02/friends.jpg...\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==\" id=\"dimg_35\" data-deferred=\"1\" data-iml=\"1585523892832\"></div></a><div class=\"mHhylf\"><div class=\"WZ5GJf\"><a class=\"qN9Ked\" href=\"https://www.google.com/search?hl=en&ie=UTF-8&ei=syyBXs2PF4nu-gT5v4DQAg&q=Friends&tbm=isch&sa=X&ved=2ahUKEwiNtdDi6MDoAhUJt54KHfkfACoQ7Al6BAgUEAw\"><button class=\"DXk5Me Qi9Fd\"><img class=\"hWHuJ\" alt=\"Arrow\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==\" style=\"max-width:24px;max-height:24px\" id=\"dimg_37\" data-deferred=\"1\" data-iml=\"1585523892832\"></button><div class=\"BNeawe ji5jpf tAd8D AP7Wnd\">View all</div></a></div></div></div></div>"
ngx.say(fix_images(htmlstr))
If you run the above code in Openresty you will see that ngx.re.gsub produces the below output. You can see that instead of doing the correct replacement at each iteration, instead it replaces only the <img src= match of the last iteration, with the <a href= capture from the first iteration. This is very bizarre and I do not understand how this has happened, but you can see that it is working correctly in the regexr.com link that I provided.
<div class="Xdlr0d"><div class="idg8be"><a class="BVG0Nb" href="/imgres?imgurl=http://www.gstatic.com/tv/thumb/tvbanners/7892603/p7892603_b_v8_ad.jpg&imgrefurl=http://google.com/search?tbm=isch&q=Friends&h=1440&w=960&tbnid=7i2_XywEZ60b2M:&q=Friends&tbnh=186&tbnw=124&usg=AI4_-kS-26wWdmwxNKHnvDZR4tXHgB8oRw&vet=1&docid=XJI53M_S0qN5PM&itg=1&hl=en&sa=X&ved=2ahUKEwjJ0_v1-MDoAhWXop4KHagGB0EQ9QF6BAgUEAQ"><div><img class="WddBJd" style="max-width:none;height:128px;max-height:128px" alt="www.gstatic.com/tv/thumb/tvbanners/7892603/p789260..." src="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" id="dimg_29" data-deferred="1" data-iml="1585528228340"></div></a><a class="BVG0Nb" href="/imgres?imgurl=https://m.media-amazon.com/images/M/MV5BNDVkYjU0MzctMWRmZi00NTkxLTgwZWEtOWVhYjZlYjllYmU4XkEyXkFqcGdeQXVyNTA4NzY1MzY@._V1_UY1200_CR93,0,630,1200_AL_.jpg&imgrefurl=https://www.imdb.com/title/tt0108778/&h=1200&w=630&tbnid=_oJOKB4azrRhBM:&q=Friends&tbnh=98&tbnw=51&usg=AI4_-kTTSv2chNcoSF0f9JBQGg3jMm2i1Q&vet=1&docid=OXs4KB9tMx4sIM&hl=en&sa=X&ved=2ahUKEwjJ0_v1-MDoAhWXop4KHagGB0EQ9QF6BAgUEAY"><div><img class="WddBJd" style="max-width:none;height:128px;max-height:128px" alt="m.media-amazon.com/images/M/MV5BNDVkYjU0MzctMWRmZi..." src="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" id="dimg_31" data-deferred="1" data-iml="1585528228340"></div></a><a class="BVG0Nb" href="/imgres?imgurl=https://static01.nyt.com/images/2020/02/21/multimedia/21xp-friends/21xp-friends-mediumSquareAt3X.jpg&imgrefurl=https://www.nytimes.com/2020/02/21/arts/hbo-max-friends-reunion.html&h=1799&w=1800&tbnid=lbyqPRixFQczNM:&q=Friends&tbnh=98&tbnw=98&usg=AI4_-kSg-e73Y4terP_KJDl-ke0ev_NTgg&vet=1&docid=dlGTY7jkGCtE_M&hl=en&sa=X&ved=2ahUKEwjJ0_v1-MDoAhWXop4KHagGB0EQ9QF6BAgUEAg"><div><img class="WddBJd" style="max-width:none;height:128px;max-height:128px" alt="static01.nyt.com/images/2020/02/21/multimedia/21xp..." src="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" id="dimg_33" data-deferred="1" data-iml="1585528228340"></div></a><a class="BVG0Nb" href="/imgres?imgurl=https://pmcvariety.files.wordpress.com/2020/02/friends.jpg?w=1000&imgrefurl=https://variety.com/2020/tv/news/friends-cast-reunion-hbo-max-1203510743/&h=563&w=1000&tbnid=7WtlGdFDXooGuM:&q=Friends&tbnh=98&tbnw=174&usg=AI4_-kQFuG5qdui9QRsqjY7Kq4unEosqSA&vet=1&docid=UtMcQG_8oL6A0M&hl=en&sa=X&ved=2ahUKEwjJ0_v1-MDoAhWXop4KHagGB0EQ9QF6BAgUEAo"><div><img class="WddBJd" style="max-width:none;height:128px;max-height:128px" alt="pmcvariety.files.wordpress.com/2020/02/friends.jpg..." src="http://www.gstatic.com/tv/thumb/tvbanners/7892603/p7892603_b_v8_ad.jpg" id="dimg_35" data-deferred="1" data-iml="1585528228410"></div></a><div class="mHhylf"><div class="WZ5GJf"><a class="qN9Ked" href="https://www.google.com/search?hl=en&ie=UTF-8&ei=oj2BXsn6NpfF-gSojZyIBA&q=Friends&tbm=isch&sa=X&ved=2ahUKEwjJ0_v1-MDoAhWXop4KHagGB0EQ7Al6BAgUEAw"><button class="DXk5Me Qi9Fd"><img class="hWHuJ" alt="Arrow" src="data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" style="max-width:24px;max-height:24px" id="dimg_37" data-deferred="1" data-iml="1585528228340"></button><div class="BNeawe ji5jpf tAd8D AP7Wnd">View all</div></a></div></div></div></div>
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the provided Lua example through the ngx.re.gsub entry point with the stated OpenResty version and HTML input. Compare the expected four replacements with the actual output, then trace the substitution behavior to determine whether the discrepancy comes from the module or the supplied pattern; done means the reported behavior is explained and the appropriate project change is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- lua, nginx
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100