? Editing: Post:21.body Save Delete Cancel
Content changed Sign & Publish new content

BinChan's ZeroNotes

Messy Notes about using ZeroNet and other things
All code published here are under CC-BY-SA licence, if not compatible, then MIT license X)

Follow in NewsfeedFollowing

TOC by date
TOC by tag
View in light theme

Latest comments:

Add new post

Title

21 hours ago · 2 min read ·
3 comments

tag:
Body
Read more

Not found

ZeroMe Feed Bot

on Apr 12, 2017
tag: bot zerome

This is a python script that will turn a ZeroMe Profile into a RSS feed reader.

RSS (Rich Site Summary; originally RDF Site Summary; often called Really Simple Syndication) uses a family of standard web feed formats to publish frequently updated information: blog entries, news headlines, audio, video. An RSS document (called "feed", "web feed", or "channel") includes full or summarized text, and metadata, like publishing date and author's name. Src:Wiki

How to Subscribe to RSS Feeds

How to Install:

  1. Backup your /path/to/zeronet/data/users.json, delete it and restart ZeroNet. Create a new ID (ZeroID, KaffieID, etc).
    Clone a hub (Don't mess up existing hub, newcomers will be disappointed when they find only Robots and Zombies in the public timeline. Also you can control your post storage limit with your own hub, or suffer this content.json: Include too large 277327 > 50000 ... Sign failed: Invalid content XD), then login ZeroMe with your new ID and your hub.

  2. Create a folder named FeedBot under your ZeroNet root directory:
    $ cd /path/to/zeronet && mkdir FeedBot
    Download the script file ↗ or copy it from Nullpaste to the folder, also under FeedBot folder create an empty file named with rssAddresses.txt.

    PS: One-line command: $ cd /path/to/zeronet && mkdir FeedBot && cd FeedBot && touch rssAddresses.txt && wget https://git.gnu.io/chbinnc/Userscripts/raw/master/FeedBot/feedbot.py

  3. Add your feeds:
    There are two way to add your feeds:
    One way is to directly copy your feed url to rssAddresses.txt, one url one line.
    The other way is to create post with your bot ZeroMe account. The format is add-feed YOUR_FEED_URL. Your can also delete your feed with del-feed URL. That's cool, isn't it ;) You can also use your major ZeroMe account to add and delete feed url. Check the code to add the feature by yourself :P

  4. Backup /path/to/zeronet/data/users.json (You need to copy your bot site address, id auth address and auth key later when first time running the script), restore your major users.json and restart ZeroNet.

  5. Run the script:
    Install package dependency feedparser: $ pip install feedparser or with other method. Then simply run it with $ python feedbot.py. If you want to run it periodically, you can use $ watch -n TIME_INTERVAL_IN_SECOND -x python feedbot.py. The script normally won't fetch anything if there are no new content. But some websites don't support etag and last-modified header, the script has to always fetch feeds from them, but still won't update anything.
    PS: On my Raspberry pi 2 watch doesn't work together with tmux, so I use this one instead: $ while sleep TIME_INTERVAL_IN_SECOND; do python feedbot.py; done.

Screenshot:

Screenshot
Bildschirmfoto_2017-04-14_09-42-10.png (24x24)

Bugs and Troubleshooting:

1. The script sometimes hangs up.

Official feedparser package doesn't support timeout setting, so you need to install it from another branch. How to:

  1. Uninstall the package if you've installed it: $ pip uninstall feedparser.
  2. Clone the branch with timeout function and install it (Default is 30 seconds, check this): $ git clone https://github.com/JPFrancoia/feedparser && cd feedparser && python setup.py install --user

Note: There is another solution by using requests library instead [1] [2], I will check it later^TM if I have time.

Todo:

  • Also add Feed Summary (300-500 character limit).
  • Keyword filters especially for large websites.
  • Remove posts with keyword
  • Set limit of post size
  • Any recommendation?

License:

MIT

27 Comments:

user_name1 day ago
Reply
Body
schiz0on Jan 05, 2019
Reply

I had unauthorized access to my ID that I created to use the feedbot

schiz0on Nov 14, 2017
Reply

binchan2: ... if you set size limit for the feed bot, you need to change the code for id and next_id...

DONE

thanks

binchan2on Nov 14, 2017
Reply

schiz0:

Sorry, because there's a big bug when I updated zeromessage in my blog, your profile data was lost. But I keep backup my data everyday, so it's recovered now.
Also if you set size limit for the feed bot, you need to change the code for id and next_id, so some feeds may have the same id and won't show up:

            bot_data['post'] += [{
-                    'post_id': len(bot_data['post']) + 1,
+                    'post_id': bot_data['post'][-1]['post_id'] + 1,
                    'body': '## [' + entry['title'] + '](' + entry['link'] +')' + '\n\n | ' + \
                            entry['summary'] + '\n\n-- [' + entry['feed_title'] + '](' + \
                            entry['feed_link'] + ')' + ' ' + entry['date'],
                    'date_added': int(time.time())
                    }]
-        bot_data['next_post_id'] = len(bot_data['post']) + 1
+        bot_data['next_post_id'] = bot_data['post'][-1]['post_id'] + 1
schiz0on Nov 08, 2017
Reply

binchan2: ... this works for me:..

DONE!

I know, you've already suggested this solution in a previus message, but I don't know why, I've solved my problem only now

anyway,
thank thank thank you

now I can schedule your scripts in cron!

binchan2on Nov 08, 2017
Reply

schiz0: wow! I'm impressed
now I've only one problem:when I start my FeedBot for ZeroMe I've to put my "privatekey" manually, to sign news; it could read this variable from a file

Thx :D Although I don't have the privatekey problem for my ZeroMe bot, I suffered it when developing ZeroTalk bot, this works for me:

        sitesign.stdin.write(AUTH_KEY)  # https://stackoverflow.com/questions/16768290/understanding-popen-communicate
        #sitesign.communicate(input=AUTH_KEY)[0]
schiz0on Nov 08, 2017
Reply

binchan2: I add this before # Sign and publish content to limit the size for...

wow! I'm impressed

now I've only one problem:
when I start my FeedBot for ZeroMe I've to put my "privatekey" manually, to sign news; it could read this variable from a file

binchan2on Nov 03, 2017
Reply

schiz0: it could be useful a sort of function to delete post older than a selectable date

I add this before # Sign and publish content to limit the size for my ZeroTalk dev bot:

        while os.path.getsize(DATA_JSON) > 48000:
            bot_data = json.loads(open(DATA_JSON).read().replace("\\u0000", ""))
            del(bot_data['topic'][1:11])
            json.dump(bot_data, open(DATA_JSON, "w"), indent=2)
        # Sign and publish content

If you want to use it for ZeroMe bot, change 'topic' to 'post'. I think setting a specific date won't reduce the size, if you prefer to set an interval, you need to change the code a bit.
Size limit of ZeroTalk is 50000, but when I set it to 50000 in the code above, it still happens that the size is a little larger than 50000, so I set it to 48000.

schiz0on Nov 03, 2017
Reply

it could be useful a sort of function to delete post older than a selectable date

schiz0on May 08, 2017
Reply

binchan2: It works for me with both Python 2 and Python 3. What kind of system are you using?

Python 2 on Linux

Maybe the relative code siteSign.communicate(input=AUTH_KEY)[0]

this line is commented on my downloaded script,
the active one is siteSign.communicate(AUTH_KEY),
but I've also tried to invert the comment (uncomment the first)

... You can check this question for some ideas

I'll take a look

(I want to cron the script)

binchan2on May 06, 2017
Reply

schiz0: I've tried, without success, to add AUTH_KEY in the Signing section

It works for me with both Python 2 and Python 3. What kind of system are you using? Maybe the relative code siteSign.communicate(input=AUTH_KEY)[0] only works on Linux. You can check this question for some ideas

schiz0on May 04, 2017
Reply

schiz0: when I run the script after the parsing process it ask me the private key before publishing, so I've to put it manually

I've tried, without success, to add AUTH_KEY in the Signing section

schiz0on May 03, 2017
Reply

in feedOutput I've added these lines

  • 'author': entry.author,
  • 'feedtitle': rssfeeds.feed.title,

and this

'body': '#[' + entry['title'] + '](' + entry['link'] + ')\n\n' + '[*' + entry['feedtitle'] + ' - ' + entry['author'] + '*]\n\n' + entry['summary'],

is the output formatting line

schiz0on May 02, 2017
Reply

binchan2: I guess you mean a function that easily copy and paste your bot post to your major ZeroMe account :3

exactly so

binchan2on Apr 28, 2017
Reply

schiz0: now we need a ZeroMe's "reblogging" function
;-)

I guess you mean a function that easily copy and paste your bot post to your major ZeroMe account :3

schiz0on Apr 28, 2017
Reply

now we need a ZeroMe's "reblogging" function

;-)

schiz0on Apr 28, 2017
Reply

binchan2: ... Some item's descriptions may have very long text and heavily formatted in html...

I like this

You can try it yourself. In feedbot.py uncomment import html2text at line 6 and 'summary': html2text.html2text(entry.summary) at line 71. Then add summary to line 117 like 'body': entry['title'] + '\n' + entry['link'] + '\n' + entry['summary'],

I've formatted the output with this line
'body': '**[' + entry['title'] + '](' + entry['link'] + ')**\n\n' + entry['summary']

>

thank you very much

binchan2on Apr 28, 2017
Reply

schiz0: wooow!it works!
:)
but it could be better if it publish also the item's description... no?

Glad to hear that it works for you :D

Some item's descriptions may have very long text and heavily formatted in html. I guess it's not easy convert them correctly into markdown. Add some plaintext won't be a hard work. But still need some time to debug.

You can try it yourself. In feedbot.py uncomment import html2text at line 6 and 'summary': html2text.html2text(entry.summary) at line 71. Then add summary to line 117 like 'body': entry['title'] + '\n' + entry['link'] + '\n' + entry['summary'],

schiz0on Apr 28, 2017
Reply

when I run the script after the parsing process it ask me the private key before publishing, so I've to put it manually

schiz0on Apr 28, 2017
Reply

wooow!
it works!

:)

but it could be better if it will publishes also the item's description... no?

binchan2on Apr 28, 2017
Reply

schiz0: 1) ... 'login ZeroMe with your new ID and your hub'...How can I select the hub? (I see only the link to change the user)

I think the information is also stored in users.json, so if you have backed up your old users.json, delete it from /path/to/zeronet/data and restart ZeroNet, when you get on ZeroMe again, it will first ask you to choose an ID, then ask you to choose a hub.

schiz0on Apr 28, 2017
Reply

done:

  • the script give me these errors:
    Traceback (most recent call last):
    File "/home/schizo/Scrivania/ZeroBundle/ZeroNet/FeedBot/feedbot.py", line 107, in <module>
    feedOutput = getFeedAndRenewAddr(addressandetag)
    File "/home/schizo/Scrivania/ZeroBundle/ZeroNet/FeedBot/feedbot.py", line 67, in getFeedAndRenewAddr
    unix_time = int(time.mktime(entry.published_parsed))
    File "/usr/lib/python2.7/dist-packages/feedparser.py", line 424, in __getattr

    raise AttributeError, "object has no attribute '%s'" % key
    AttributeError: object has no attribute 'published_parsed'

  • when I publish content with my new ID I've a publishing error

  • now there's my hub listed on ZeroHello (and in the hubs list of KaffieHub), but I don't want to register new users, how can I delete it from the "MERGED: ZEROME" list?

schiz0on Apr 28, 2017
Reply

1) ... 'login ZeroMe with your new ID and your hub'...
How can I select the hub? (I see only the link to change the user)

binchan2on Apr 27, 2017
Reply

schiz0:

in point 4... I've to put the old 'users.json' (of point 1) in '/path/to/zeronet/data/' (of the first installation)?

Yes, those steps are just to create another ZeroMe profile for this script.

when I 'Create a new ID' what about my old one? Have I to change my ZeroMe Hub?

You old ID is in the old 'users.json', so when you put it back in point 4 and restart ZeroNet (I forget to mention this), you will again login with your old ID. Also you don't have to change your ZeroMe Hub.

I'm sorry for my lamer's questions

I'm not a native English speaker too, so don't mind :)

schiz0on Apr 27, 2017
Reply

I'm sorry for my lamer's questions:

  • in point 4... I've to put the old 'users.json' (of point 1) in '/path/to/zeronet/data/' (of the first installation)?
  • when I 'Create a new ID' what about my old one? Have I to change my ZeroMe Hub?
binchan2on Apr 22, 2017
Reply

schiz0: I wish a better installation, but it's very interesting

Thanks! If ZeroMe supports multi-user, I guess the installation and user experience of this script will be better. But still I add a one-line command to make step 2 a bit simpler.

schiz0on Apr 20, 2017
Reply

I wish a better installation, but it's very interesting

p2pon Apr 14, 2017
Reply

cool!

Select user
  • Welcome to ZeroMessage!
This page is a snapshot of ZeroNet. Start your own ZeroNet for complete experience. Learn More