regex - robot.txt syntax not understood -


i getting error on first line of robot.txt user-agent: *

my robot.txt follows:

user-agent: * disallow: /search/ disallow: /_layouts/  disallow: /blog/_layouts/ disallow: /blog/_layouts/ disallow: /reusablecontent/ disallow: /reports%20list/ disallow: /workflowtasks/ disallow: /sitecollectionimages/ disallow: /documents/forms/ disallow: /pages/forms/ disallow: /internet/

i have figured there space /reports%20list/. creating issues? not sure. appreciated.

the trouble here 3 invisible bytes @ beginning of utf-8 formatted robots.txt file, called bom (byte order mask). bom optional. yes, need file utf-8 formatted, google not optional bom in expected clear , simple robots.txt file , parsing fails.

you can read more detailed explanation here.


Comments

Post a Comment

Popular posts from this blog

user interface - Python attempting to create a simple gui, getting "AttributeError: 'MainMenu' object has no attribute 'intro_screen'" -

complete newbie here python started learning language few days ago (with no programming experience beforehand). i'm bashing skull against desk here, trying create 1 menu button lead menu, supposed replace/hide/destroy previous menu (either works, long process can reversed). what i've come far: import wx class mainframe(wx.frame): def __init__(self): wx.frame.__init__(self, none) self.centre() self.main_menu = mainmenu(self) self.intro_screen = introscreen(self) self.intro_screen.hide() class mainmenu(wx.frame): def __init__(self, parent): wx.frame.__init__(self, parent=parent) self.main_menu = mainmenu panel = wx.panel(self) sizer = wx.boxsizer(wx.vertical) nextscreen = wx.button(panel, label='next screen', size=(150,30)) nextscreen.bind(wx.evt_button, self.nextscreen) sizer.add(nextscreen, 0, wx.center|wx.all, 5) self.show() self.c...
Read more

jquery - Common JavaScript snippet to share files on Google Drive, Dropbox, Box.net or SkyDrive -

this first assignment of file sharing. need share files our website above mentioned locations. currently, have been exploring each service’s api file sharing. question 1 : there javascript code snippet available (like addthis.com ) share files? question 2 : should use server side coding file sharing or javascript advisable? thanks viewers. found useful plug-in named "ink file picker". saves given http file resource directly users online storage engine. url: https://developers.inkfilepicker.com/docs/web/ live example: http://ctrlq.org/save/
Read more

Android Gson.fromJson error -

i've completed tutorial how connect json url , parse content using gson library. code works fine url specified in tutorial, tryed modify work this url . tryed display 1 field second ulr, base field, can't work. here code, please help: public class jsonparsingactivity extends activity { string url = "http://api.openweathermap.org/data/2.5/weather?q=london,uk"; @override public void oncreate(bundle savedinstancestate) { super.oncreate(savedinstancestate); setcontentview(r.layout.main); inputstream source = retrievestream(url); gson gson = new gson();// main gson class reader reader = new inputstreamreader(source); searchresponse response = gson.fromjson(reader, searchresponse.class); //gson.fromjson(string, class) toast.maketext(this, response.base, toast.length_short).show(); } private inputstream retrievestream(string url) { defaulthttpclient client = new defaulthttpclient(); httpget getrequest = new httpget(url);...
Read more