Search for words with Scandinavian characters

Newbie Joined: Nov 03, 2007 Posts: 88

When I use Search to search for an article with a word with Scandinavian letters it works normally until I hit Next results, which gives a page without any search results. If Shortlinks is disabled it works very well and there is no problem.
This is the link that gives no search results:
http://www.mysite.com/search.html&author=&topic=0&min=10&query=tv%E5&type=stories&category=0

Newbie Joined: Nov 03, 2007 Posts: 88

I have been thinking on the problem and I think my problem here is caused by RewriteRule for this in .htaccess. The Scandinavian letters causing this is åäö, which comes after z in the Swedish alphabet.

This Rewriterule is probably making this:

RewriteRule ^search-([a-zA-Z0-9]*)-([0-9]*)-([0-9]*)-([/:\-\'{}()\,\._&a-zA-Z0-9+=\ ]*)-([a-zA-Z]*)-([0-9]*).html modules.php?name=Search&author=$1&topic=$2&min=$3&query=$4&type=$5&category=$6 [L]

I have modified:

-([/:\-\'{}()\,\._&a-zA-Z0-9+=\ ]*)-([a-zA-Z]*)-

to:

-([/:\-\'{}()\,\._&a-zA-Z0-9+=\å\ä\ö\ ]*)-([a-zA-Z]*)-
did not make any difference
-([/:\-\'{}()\,\._&a-öA-Ö0-9+=\ ]*)-([a-zA-Z]*)-
gave Internal server Error
-([/:\-\'{}()\,\._&a-zA-Z0-9+=\ ]*)-([a-zA-Z\å\ä\ö\]*)-
did not make any difference

The result of normal function of this link should be like this:

http://www.mysite.com/search--0-10-vara-stories-0.html

vara is the word that is put in the Search box.

Site Admin/Owner Joined: Feb 12, 2005 Posts: 952

1. I need to know if the link is being shortened properly (i.e., when you view the link in the browser status bar, is it correct)?

2. Or, is the link appropriate, but when clicked you get a "404 Not Found" type error.

Just a side-note, one could also try using [[:alnum:]] instead of the [a-zA-Z\å\ä\ö\] and see if that works.

Newbie Joined: Nov 03, 2007 Posts: 88

montego wrote:

1. I need to know if the link is being shortened properly (i.e., when you view the link in the browser status bar, is it correct)?

2. Or, is the link appropriate, but when clicked you get a "404 Not Found" type error.

Just a side-note, one could also try using [[:alnum:]] instead of the [a-zA-Z\å\ä\ö\] and see if that works.

1. The link is not shortened at all. This is the link, when I click on it:

http://www.mysite.com/search.html&author=&topic=0&min=10&query=tv%E5&type=stories&category=0
In the status bar it is:

http://www.mysite.com/search.html&author=&topic=0&min=10&query=två&type=stories&category=0

2. I get no error, when I click on the page. The searchbox is displayed, but there are no displayed search results beneath it.

I tried [[:alnum:]] but it did not make any difference. This is how I did it:

([[:alnum:]]*)-([[:alnum:]]*)-

Site Admin/Owner Joined: Feb 12, 2005 Posts: 952

You need to make sure and change the GT-Search.php script too under the ShortLinks directory in order to get the URL to be shortened.

I tested your

[a-zA-Z\å\ä\ö\]

and it definitely will not work. Try using this instead:

[a-zA-Zåäö]

You do not need to escape the additional characters.

Again, do not forget to change both GT-Search.php AND .htaccess to match.

Site Admin/Owner Joined: Feb 12, 2005 Posts: 952

By the way, I also just now tested the following and it worked as well:

([/:\-\'{}()\,\._&[:alnum:]+= ]*)

See where/how I placed the [:alnum:] directive?

Newbie Joined: Nov 03, 2007 Posts: 88

Many thanks!
I have added as you suggested:
-([/:\-\'{}()\,\._&a-zåäöA-Z0-9+=\ ]*)-([a-zA-Z]*)-

and also åäö to GT-Search.php and it is working correct now.

Site Admin/Owner Joined: Feb 12, 2005 Posts: 952

Had you tried my [:alnum:] approach? Just curious as since I test mostly in english, it would be nice to know if what also solves the issue because I am hoping it takes your web site character set into consideration.

Newbie Joined: Nov 03, 2007 Posts: 88

[:alnum:] worked, if I added it in .htaccess, but in GT-Search.php I had to have ;query=([a-zåäöA-Z0-9]*) to get it to work, because [:alnum:] did not work for search words with åäö in this file.

I think that people in Denmark, Norway and Germany can have similar problems like this.

Site Admin/Owner Joined: Feb 12, 2005 Posts: 952

Ok. Good to know. That is what I was concerned about. The PHP functions themselves are PCRE based and not straight Perl and so that is not working.

Thank you for testing that out for me!

Newbie Joined: Nov 03, 2007 Posts: 88

BTW, it is a very similar problem with the Search of the Web_Links module, that is solved in almost identical fashion.

Newbie Joined: Nov 03, 2007 Posts: 88

I have a strange problem with Search and Shortlinks.

If I search for the Swedish word ångest I get a lot of hits more than 30. With Shortlinks off everything is normal and I can browse the next 10 hits etc. The link is encoded in ISO-8859-1.

With Shortlinks on I can't browse next 10, because suddenly the link seems encoded in UTF-8 and I get 404 error when trying to click on it.

In the index.php I have added the line:

$query = rawurlencode($query);

just above this code:

$next=$min+$offset;
if ($x>=9) {
print '<br /><br /><center><a href="modules.php?name='.$module_name.'&author='.$author.'&topic='.$t.'&min='.$max.'&query='.$query.'&type='.$type.'&category='.$category.'">';
print '<b>'._NEXTMATCHES.'</b></a></center>';
}
}

The link then gets encoded in ISO-8859-1 but when I click on it I don't get next results, but just the Search page without any results.

This is the link with Shortlinks off:

http://www.mysite.com/modules.php?name=Search&author=&topic=0&min=10&query=%E5ngest&type=stories&category=0

This is the link with Shortlinks from URL:
http://www.mysite.com/search--0-10-ångest-stories-0.html
If I copy the link I get this:
http://www.mysite.com/search--0-10-%C3%A5ngest-stories-0.html

Site Admin/Owner Joined: Feb 12, 2005 Posts: 952

Well, definitely do not do the url encode. I would comment that out and try adding in the '%' sign into the rewrite rules. The reason I saw this is look at the link when ShortLinks is off. Its already encoded. You just need to make sure the encoding isn't messed up when ShortLinks is turned on.

Newbie Joined: Nov 03, 2007 Posts: 88

I have tried with % in the rewrite rules, but it did not make any difference. I also discovered that I had made a mistake in the rewrite rules that made rawurlencode to not work. When I corrected this mistake rawurlencode did work actually. I have added it on two places approx on line 272 and 279 just before next and previous.

Site Admin/Owner Joined: Feb 12, 2005 Posts: 952

Ok, will have to keep this in mind for further RN work. Thanks!

So, this rawurlencode has completely solved your issue then?