SadLoli
sadloli@zeroid.bit

A random ZeroNet user, now motivated by the shock of my life to better the things I love!


SadMetadata(in development)

A backup of all metadata on sadpanda/exhentai searchable by gallery url.

Help distribute this user's images
Mute sadloli@zeroid.bit

Activity feed

SadLoli commented on Port220irq5dma1's post: @Styromaniac: anime IsReaL?
SadLoli started following Port220irq5dma1
SadLoli commented on Lizardricon's post: @ulrichard: >All the random non-monetary stuff I saw so far was in text form, maybe even ascii art. In the thread I referenced, they were talking about that there already is an ("really tame") image in there: >"That image has a very low resulution and depicts a girl(she looks to be around age 15-17) flashing her boobs to the camera. Thats it. " There is also the transactionID/hash to it in there. >In what format would that be? The README in [the repo](/126KYGEVcPaw9u7NKA6ZxqBUgFjGxqpF15) for it is pretty elaborate. >"The file is cut into 20 byte slices. These are than encoded as retrieving addresses and put into an Bitcoin transaction." In there is also a more technical explanation. @caryoscelus: >Such program would essentially contain some predefined CP picture. Like the negative of a photograph. The actual information would've been encoded into source code. In this case the actual information is encoded into the blockchain.
SadLoli commented on Lizardricon's post: @Jaoheah: I mean you could, but someone could just port the program to put it into that other blockchain. It's more of an issue of the immutability of blockchains, which is fundamental for all cryptos, rather than the individual blockchain. @caryoscelus: Doesn't look like anyone had the balls to actually go through with it (even within the thread). But with the influx of people of questionable morals, I'm not sure if that it will keep being that way. News like this would fly through the media, if that were to happen.
SadLoli commented on Lizardricon's post: >I know Bitcoin will be something With the surge in popularity of zeronet, I think bitcoin will be dead within months, because of [this millchan /b/ thread](/1ADQAHsqsie5PBeQhQgjcKmUu3qdPFg6aA/?:users/1H5YxVHvcfKM4STjJwyjYaBe8qbPyR6EZr:b:9ea42cb5-fb15-4de0-a8c3-b7f25fb1af0a), which plans to have childporn inserted into the blockchain, thus making it illegal to download it.
SadLoli commented on Optic's post: Are you sure?
SadLoli commented on SadLoli's post: @Zbroevich: Me neither, only getting timeouts.
SadLoli commented on SadLoli's post: @Zbroevich: Last I saw was in the 8chan /v/ thread. ctrl+f type 'zeronet' and you should find it. If you can still access 8chan, that is.
SadLoli commented on SadLoli's post: @SSDifnskdjfnsdjk: >millchan engine Yeah, looked through the repo of it, too. Apparently the development on the moderation system was started because of it. It looks like millchan engine was actually rather kind of an PoC, because many of the commits in the repo break compatibility with the current hubs/boards and it really isn't up to date with the code in the repo.
SadLoli commented on SadLoli's post: @SSDifnskdjfnsdjk: >unsure if board owner can give mod rights also to others and how it is done. I've gone through the source code and they can't.(but that is in development, though) >or other zites (maybe like this one) 'Splash damage' was my main concern. On millchan you can just mute the board and you are done. Having these retards leak out to everywhere else will be annoying.
SadLoli commented on SadLoli's post: @SSDifnskdjfnsdjk: Done.
SadLoli commented on SadLoli's post: @Wormstweaker: Judging by what @Sirlotaplot: said [here](/Me.ZeroNetwork.bit/?Post/1MoonP8t4rk9QamBUPh5Aspkwa1Xhf5ux2/1PBCaxayoEqvXFuayjxzGJvPub3BMUJUJT/1564505863), maybe.
SadLoli commented on SadLoli's post: @Wormstweaker: Holy shit your right!!
SadLoli commented on SadLoli's post: @Kid Courageous: Took me a while, but I have now made a [GitCenter repo](/1MqTXukdSULZVC6kXVwgC7pNkUum6TbmyH). @all here: I'm making a Contributors/Thanks page if you want to be listed there under the 'Information about zeronet' section please tell me that. I will not add anyone that didn'e explicitly told me they want their name on there. @Zboy: I'm getting some weird crypt errors when I visit ZeroMail... Tell me, if you already sent me anything!
SadLoli commented on SadLoli's post: @caryoscelus: Thanks! Gonna look into those later.
SadLoli commented on SadLoli's post: @Zboy: I think I already did. Can you send me a mail to test if it worked? @Mg0: I'm glad you helped!
SadLoli commented on SadLoli's post: @blurHY: Looking forward to that.
SadLoli commented on SadLoli's post: @SSDifnskdjfnsdjk: >Wondering if also images will be shared (somewhere, some day). I am however thinking about including the gallery thumbnails, since the guy that made the front page scrape this data is based on also made a collection of every thumbnail for each gallery that has been scraped. But full on galleries probably not on that site, but I plan to do that eventually.
SadLoli commented on SadLoli's post: @SSDifnskdjfnsdjk: Just pushed an (couple of) update(s). You can now just push an button and it should download them automatically. I also reworked the structure I stored the json files such that every content.json file now only stores about 1000 files. Signing page updates now works **much** faster for me since the database doesn't need to be rebuild every god damn time.
SadLoli commented on SadLoli's post: @blurHY: Sounds sweet! When will this be ready? I thought about doing this on IPFS but decided this is more user friendly.
SadLoli commented on SadLoli's post: @weakish: All json files should be around 300kB or less. So a limit of 1MB should be fine. @Wormstweaker: >My 0net instance crashed a few times, but that's probably because I installed it like shit Seems to be an issue of zeronet every time a database rebuild happens with a fair amount of data maybe what I written [here](/Me.ZeroNetwork.bit/?Post/1MoonP8t4rk9QamBUPh5Aspkwa1Xhf5ux2/1PBCaxayoEqvXFuayjxzGJvPub3BMUJUJT/1564505863) in response to @caryoscelus: has something to do with it. @nofish: Is that my fault or a known bug in zeronet? >Anyway, files downloaded, and seeding 👍 >somehow I downloaded 627mb, instead of the 622 announced Odd. Maybe that is the total amount downloaded and counts failed downloads and webside data(html, js...) as well? >so it checks on eh if the gallery still exist, even if the metadata isn't in the database I would need to make a XHR connection to clearnet to that and I'n not sure if that works. Need to do some testing on that, but sure if it works.
SadLoli commented on SadLoli's post: @SSDifnskdjfnsdjk: >Downloaded: 0.00MB Oh okay, I haven't done anything to make the site download the pieces it needs, yet. So unless you press the Download and help distribute button, nothing will be downloaded. Therefore the database is empty therefore every query will return no results. You can check this by running: 'select count(ID) as TotalNumberOfGalleries from metadata' (without the quotes) in the 'Make your own Query here!' page and see how many galleries your database currently has. >"(..db.|nbproject.|\..)" That means that my zeronet client will not sign and publish the database that my client build for me. 'nbproject' isjust the project folder of netbeans(the IDE I use) and '\..'tells zeronet to no sign any hidden files. @blurHY: Yeah, I think I'm gonna sort the jsons by ID rather than in a pack of 600 per file, maybe this helps.
SadLoli commented on SadLoli's post: @Mg0: Good to know, I guess I'm going to do some testing as well. @SSDifnskdjfnsdjk: Runs fine for me. It feels like 500MB shouldn't be too bad for a database. Everyone with a HDD running into issues? The search can currently only search for galleryIDs. That's why there's the beginning part of the url right in front of the search field. If you have the link to a gallery you saved, and want to know the metadata for just complete the link and hit search. You can put in '/g/1330079/ea3aaac89f/', a personal favorite of mine ;-), to test it if you like. @blurHY: >wasting resources in what way?
SadLoli commented on SadLoli's post: @Wormstweaker: But it id download in the end, right?
SadLoli commented on SadLoli's post: @Zboy: I do now!
SadLoli commented on SadLoli's post: @Kid Courageous: So [GitCenter](/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t)?
SadLoli commented on SadLoli's post: @Tooltoulto: Greatly appreciated! (please tell me if it worked)
SadLoli commented on Tooltoulto's post: @Tooltoulto: Well, I'm beat. People only talked about this being specific to Tor Browser.
SadLoli commented on Tooltoulto's post: I also noticed that. It seems to be a bug/feature in Tor Browser. If you use firefox or chrome it works.
SadLoli commented on SadLoli's post: Here is what I have so far: http://127.0.0.1:43110/19dfPUtRhw6KxzAYBzfntburwuZDQULMZ3 (Is it working?)
SadLoli commented on SadLoli's post: @caryoscelus: >Hmm, i'm not sure what exactly is your purpose. For now just building a site where one can search: - by SQLite Query - by metadataID - by multiple metadataIDs - by browsing through the data So that I lean how do zeronet, basically and at some point basically this: >build a site on which users can upload whatever they have downloaded and then display it as it was intended And looking at what @Sirlotaplot: said, that'll probably either be a fork of 1+Gallery or a collaboration with it since 1+Gallery is based of 0Gallery which itself was based of sadpanda/exhentai. Doesn't make much sense to reinvent the wheel here. >What is the shape of metadata you've got? I'm not sure what you mean by 'shape'. Metadata object in js after a query: { "title": String, "pages": Integer, "uploader": String, "href": String, "type": String, "uploadDate": Integer, "tags":[ { "namespace":String, "name":String, "power":String }, {...} ], "SET_INFO": String, "ID": String, "PAGE": Inetger, "IS_DATA_MISSING": Boolean } I'm currently signing what I have so far. And I've noticed that it takes a lot longer if I include a dbschema.json that will actually use the data to make a database. I think ZeroNet is rebuilding the database regardless of whether or not it needs to be rebuild on every signing. The rebuild happens between: [log]- css/index.css and [log]Adding timestamp and sha512sums to new content.json
SadLoli commented on SadLoli's post: @caryoscelus: >Nah, i meant what's the distribution terms / license? Were those author-uploaded pics, "illegal" copies, or just anything? That thing is difficult: Most stuff on there is commercial, but only really available in japan and in japanese. However some of the stuff was also from western artists. Most artists didn't mind it being uploaded there, while some (historically) did. Some artists (both western and eastern) used it to upload their own works on there. Many didn't. **A lot** of translators published their works(translations/edits like decensoring and many more) there. Some had their translations uploaded there of which some didn't want them to be there. So in short: kind of? Strictly answering the that question is inconclusive at best: Were those author-uploaded pics, "illegal" copies?: No, except when they were. You could think of sadpanda as a weaker enforced pixiv, if that helps. >AFAIK ZeroTalk and here are two most active places.. Good to know. >split by some thematic criteria and allow users to download only parts of db they're interested in The main purpose would be to search by id, which look like this: '/g/1330079/ea3aaac89f/'. Because most people have only downloaded the images, but not the metadata. Not sure what I can do there in terms of presorting
SadLoli commented on SadLoli's post: @Kid Courageous: Sweet! Should I make a repo somewhere and how should be communicate? I've already seen that there is GitCenter, are there others on here?
SadLoli commented on SadLoli's post: @nekololi: >but would be happy to help seed as well as link any site that gets made in the process on loliguide. Highly Appreciated! >loliguide Nice! >i don't know offhand if there is anyone with zeronet development experience who is also interested in a hentai related project like this If you happen to find someone that does please point them in this direction. Millchan /l/ seems dead, is there somewhere else a more active community here on zeronet?
SadLoli commented on SadLoli's post: @Mg0: I guess proxies are a good poin't since they, if commercial servers, will have fast SSDs to quickly query the database. >just make sure to use content.json files properly. Not too many files for each How many would you say that would be?
SadLoli commented on SadLoli's post: @SSDifnskdjfnsdjk: >zeronet is not yet redy for large databases The database worked fine for me(on SSD), it's ~530MB not sure if that's much. >Btw. i would like to help you, but as mentioned, i am not a developer nor a coder. Dude, you are already helping a ton, as is everybody in this thread <3
SadLoli commented on SadLoli's post: @Sirlotaplot: >blurhy's IPZN or something like it Look interesting. >I'd bet it can handle 10x what I'm doing now (if it's linear), but that'd still only be 4TB. That's good to know so we could split the ~50TB into like 13 Hubs and see from there. There's also the possibility that @1pgallery manages to improve efficiency so much that we don't need the hub thing, but I guess we just have to see for now. >Engine one has each board be it's own zite which means you only download the ones you want @SSDifnskdjfnsdjk: >Millchan Engine is utilising merger sites - using "merger" feature of the zeronet. Oh, okay thanks you two!
SadLoli commented on SadLoli's post: @caryoscelus: >I don't think 0net can work with compressed JSON I didn't mean compressed as in gzip/tar compressed, but compressed as in no indentation/spaces in the json file(with those it's about 1.3 or 1.8 GB in size). >Perhaps it might be a good idea to split it, though Absolutely the full file took forever. to sign I've split it into about 3000 files with 600 entries each. It works reasonably well, database building takes ages however. But I'm not sure if I can do anything about that. >It is also possible to not use 0net-specific stuff and just parse metadata from JS, but that might be just too slow I did the initial parsing in JS and I don't have enough fingers or toes to show you how many times the tab or even the browser crashed ; ) >Most users will likely not want to seed all of the galleries For now only the metadata. I think I'm gonna use that hub mechanic to combine it with the actual galleries. >I'm not familiar with the site See @nekololi: s comment. > but what's their status? They are down/offline, admins say that there is a full backup somewhere and the they'll look into more permanent storage options for the backup while telling people to get and save as much data to increase redundancy, so not sure how much the backup actually is a thing. >JS & 0net api (i still remember the pain of creating my meh engine..), but possibly i can help with some advice Thanks, should I make the json files optional or required? I don't think that zeronet can tell in which file the gallery metadata can be found in and querying the data only really makes sense if you have all of it to search through, right?
SadLoli commented on SadLoli's post: @Sirlotaplot: Thanks! I'm happy to get this much response from here. >very thin silver lining to the fall of Sadpanda. I guess, so. I'm actually looking forward to the discussion on p2p alternatives that will happen, once ATF gets back up. I guess that really is a plus. I've also been looking into IPFS recently, but it doesn't seem like you can have dynamic sites there, yet. Please correct me if I'm wrong, but I couldn't find anything in that regard. >I'm running two hentai manga focused ZeroGallery hubs Sweet! Currently saving space for Exhentai backups, but looks interesting. How do you reckon 1+Gallery would perform with ~50TB of shared data? \>Millchan Engine What's the difference between 'millchan' and 'millchan engine'?
SadLoli started following Sirlotaplot
SadLoli commented on SadLoli's post: @SSDifnskdjfnsdjk: Thanks, already digging through quite a bit of documentation, but more sources are always better. Millchan is still loading, I think I'm gonna make a post there a bit later, when I actually have something to show. > i seen this static torrent index zite I'll look into it. I already *really* like that you can see the source of each website here. You two seem to know quite a bit, would you or @caryoscelus: be interested in helping with development?
SadLoli commented on SadLoli's post: @caryoscelus: I have a ~530 MB compressed JSON file with all metadata for all sadpanda(with a few exceptions) galleries. I want to make a searchable site by metadata ID(the locator part of the url to the gallery) and some browsing options, in the future at least. >Perhaps the only thing needed is to make sure all those big files are marked as optional What's the difference? As for the images I thought of registering the galleries by hash(probably sha1, because that's what BitTorrent uses) and giving out gallery IDs, which will be the xored hashes of all images of the gallery. This way the order in which the images are uploaded will not matter. The site will not allow duplicates to be uploaded checked by hash. Unless one offers a higher resolution. >It seems like scrapping data from servers is a bigger task Scraping time is over, Sadpanda is dead.
SadLoli commented on SadLoli's post: @Wormstweaker: Great! I think I'll post updates here on this ZeroMe account.
SadLoli commented on SadLoli's post: @SadLoli: Why are your mentions of me linked by my mentions of you aren't?
SadLoli commented on SadLoli's post: @Wormstweaker: You don't necessarily need to have knowledge of zeronet to help. Spreading word around would also be of great value especially here, where we need seeders to remain online. Getting more people also increases the chances that someone with actual skill might choose to help. >Oh and crap, this is more of an image board I've been around, I actually prefer imageboards to reddit.
SadLoli commented on SadLoli's post: @Wormstweaker: Sweet! Gonna give it a try. Are you by any chance interested in helping?
SadLoli commented on A7oiz's post: Are you already working on something? I'm working on a browsable backup of the metadata torrent as a site here. Maybe we can team up. Do you know any good places for coordinating something like this here?
SadLoli commented on A7oiz's post: Already working on it.

Following

+
artist/gamedev
+
`ZeroNet dev.`
+
Git Center
+
26/Female - Kaffiene Developer --- [Patreon](https://www.patreon.com/aprilc0t) [Patreon Discord](https://discord.gg/hx3xXZV) [My Blog](http://127.0.0.1:43110/1A83ijw3boqTtqdLz8me7AqeK1nEK8yxeu) [My Github](https://github.com/April93) [Kaffiene Search](http://127.0.0.1:43110/1Mr5rX9TauvaGReB4RjCaE6D37FJQaY5Ba)
+
Developer of - [1+Gallery](/1PGaLLNiadMzrmw6MMMnpZnskwFcdgZoS4) ([Git](/191JBSmjUStVdF6zxqujN2MZUk1KaphEPe)) - [GalleryHub+](/14NFfVBBAFbNdqX4nyGjyBF1BRSz7jn2QP) ([Git](/114AZoCSLsZu6Ktfnfu64Uoh8pcdVYFhSX))
+
Stoner, programmer, & babysitter.
+
Random ZeroNet user
+
Browse my Hentai collection at the [Alotaplot](/1G8VWDQc5fXLkRiijJB4EZerDLmDVY75MZ) Hentai Manga & Doujinshi gallery (150+GB, 2300+). Please download and seed! Coming soon: Anthology.jp, my entire digital collection of Hentai magazines, neatly sorted with an all new interface! (240+GB, 1700+) **WARNING: Download speed is slow! But the more people use it the faster it will get, so spread the news!** Want me to add a specific manga, doujinshi or magazine? Let me know!
+
Loli is love. Loli is life. Follow me for your supply of lolis. 2D only.
+
Kawaii ZeroNet user
+
¯\_(ツ)_/¯
+
.:weird and hungry content creator:. .:lazy programmer:. .:an artificial construct of someone’s mind somewhere on the other end of the net:. - [home page](/1MCoA8rQHhwu4LY2t2aabqcGSRqrL8uf2X/) [ [blog](/1MCoA8rQHhwu4LY2t2aabqcGSRqrL8uf2X/?/blog/) ] ([clearnet mirror](https://caryoscelus.github.io)) - [my music](/1FiHm91tcDdjkiGkHZH2xoMc7Qmzrh8sv3/) - [moving pictures theatre](/1N8zibqog72CCDBDo8jrxU7C6Q8JeVND2Z/) - [short stories](/1JecUbTPHudVR3fi6KMFwQNW2mT2L8H7dq/) - [poetry](/1QDEpSBxTXiZ4ak9m9ATAVAy1SSLiRrxKs/) - [heavy bright sun](/1HAB57aQRcLHoHb3ehHrh8SZT647VWVA28/) - [Libre Dazai](/1JPGL61v5LgbfyD3pvAkCy3YqrCWqaekWH/) - [Obligatory Empty Page](/13GoB3QYyoqtN7wVpuQfgAQxn7LwucYED7/) ([repo](/1EtWCTxuY34N4C8incQgjTw8M2S2TPvZwF)) - [HydeView - yet another duct tape engine, related to Jekyll](/18JXMUqeg6u1nZXi1rYTsiUpBuzTnyi9yc/) ([repo](/1DTjNhj7Mm4hSvEQ2mEUnWp5cP1eaUzHyA)) - [donate](/1MCoA8rQHhwu4LY2t2aabqcGSRqrL8uf2X/?/donate/) (please don't donate on my 0net addresses: that might be insecure) The rumours say i may write.. - ..in a reasonably good world english - ..на достаточно литературном русском - ..外人の日本語で - ..bau la mabla lojban Please don't use automated translation to write if you don't know a language, we can use it to read instead Copyright is stupid and unethical. Free software and libre culture ftw! Being on my "following" list does not imply endorsement, though being on my "mute" list implies dis-endorsement
+
List of [popular 0net sites](/1Kt7xxVRcxHcK14Mt7YZdXfh8MzUJMkvpk/) [KopyKate videos i have shared](/big.kopykate.bit/?Channel=ssdifnskdjfnsdjk@zeroid.bit), another [KxoVid videos](/16KrKKRNHUc8WnRNuV9PgNqZQ8zqGteT4p/?/channel/1Mi7rUdBpQc99kfuvippGxgEbM7ZKdSXUD/1553291060684) [NetTalk](/1LfvE91ZF18jdG3wW62Dw7NtfTZh737KPL) computer/internet forum [ZeroNet audio library](/ZeroLSTN.bit)
SadLoli · sadloli@zeroid.biton Jul 30, 2019

Anyone here want to join the effort of building an archive of sadpanda/exhentai here on zeronet?

Where do we best coordinate a project like this? Are there already people here trying to do that and if so could someone point me in their direction?

Every help is appreciated, but I'm especially searching for someone with zeronet development knowledge right now.


Farewell Sadpanda,
see you, space cowboy...

caryoscelus · caryoscelus@zeroid.biton Aug 04, 2019Reply

@SadLoli: if you think i've been of any help, feel free to add me ;)

By the way, i've glanced at the repo and i think it's better to put js into separate .js file. Although at this size, this is somewhat aesthetic preference, i guess

SadLoli · sadloli@zeroid.biton Aug 04, 2019Reply

@SSDifnskdjfnsdjk: Done.

SSDifnskdjfnsdjk · ssdifnskdjfnsdjk@zeroid.biton Aug 02, 2019Reply

@SadLoli: i am seeding your gitcenter repo, if you wish, you can add me to the "Information about zeronet" if you think it is ok.

  • "You currently have the data of 690583 out of 690583 galleries downloaded."
SadLoli · sadloli@zeroid.biton Aug 02, 2019Reply

@Kid Courageous: Took me a while, but I have now made a GitCenter repo.

@all here: I'm making a Contributors/Thanks page if you want to be listed there under the 'Information about zeronet' section please tell me that. I will not add anyone that didn'e explicitly told me they want their name on there.

@Zboy: I'm getting some weird crypt errors when I visit ZeroMail... Tell me, if you already sent me anything!

SadLoli · sadloli@zeroid.biton Aug 01, 2019Reply

@Zboy: I think I already did. Can you send me a mail to test if it worked?

@Mg0: I'm glad you helped!

Mg0 · mg0@zeroid.biton Aug 01, 2019Reply

@SadLoli: Glad to hear I was of help.

Zboy · zboy@zeroid.biton Aug 01, 2019Reply

Can you set up Zero Mail so that I can message you?

SadLoli · sadloli@zeroid.biton Aug 01, 2019Reply

@SSDifnskdjfnsdjk:

Wondering if also images will be shared (somewhere, some day).

I am however thinking about including the gallery thumbnails, since the guy that made the front page scrape this data is based on also made a collection of every thumbnail for each gallery that has been scraped.

But full on galleries probably not on that site, but I plan to do that eventually.

SadLoli · sadloli@zeroid.biton Aug 01, 2019Reply

@SSDifnskdjfnsdjk: Just pushed an (couple of) update(s). You can now just push an button and it should download them automatically. I also reworked the structure I stored the json files such that every content.json file now only stores about 1000 files. Signing page updates now works much faster for me since the database doesn't need to be rebuild every god damn time.

SSDifnskdjfnsdjk · ssdifnskdjfnsdjk@zeroid.biton Aug 01, 2019Reply

@SadLoli: when i opened the site menu (drag & drop top right corner button) and ticked the "Download and help distribute all files" option, the count of galleries started growing and as you said i see also gallery information when i search. Wondering if also images will be shared (somewhere, some day).

SadLoli · sadloli@zeroid.biton Aug 01, 2019Reply

@SSDifnskdjfnsdjk:

Downloaded: 0.00MB

Oh okay, I haven't done anything to make the site download the pieces it needs, yet. So unless you press the Download and help distribute button, nothing will be downloaded. Therefore the database is empty therefore every query will return no results.
You can check this by running:
'select count(ID) as TotalNumberOfGalleries from metadata'
(without the quotes) in the 'Make your own Query here!' page and see how many galleries your database currently has.

"(..db.|nbproject.|..)"

That means that my zeronet client will not sign and publish the database that my client build for me. 'nbproject' isjust the project folder of netbeans(the IDE I use) and '..'tells zeronet to no sign any hidden files.

@blurHY: Yeah, I think I'm gonna sort the jsons by ID rather than in a pack of 600 per file, maybe this helps.

blurHY · blurhy@zeroid.biton Aug 01, 2019Reply

@SadLoli: Building database takes up to ten minutes

SSDifnskdjfnsdjk · ssdifnskdjfnsdjk@zeroid.biton Aug 01, 2019Reply

Database query error: No results found for'/g/1330079/ea3aaac89f/'!
-
Database (32.00kB..
-
Optional files
Downloaded:
0.00MB
Total:
622.85MB
-
main content.json has line: "ignore": "(..db.|nbproject.|\..)",
unsure what that mean, like ignore (do not download) database updates?

SadLoli · sadloli@zeroid.biton Aug 01, 2019Reply

@Mg0: Good to know, I guess I'm going to do some testing as well.

@SSDifnskdjfnsdjk: Runs fine for me. It feels like 500MB shouldn't be too bad for a database. Everyone with a HDD running into issues? The search can currently only search for galleryIDs. That's why there's the beginning part of the url right in front of the search field. If you have the link to a gallery you saved, and want to know the metadata for just complete the link and hit search. You can put in '/g/1330079/ea3aaac89f/', a personal favorite of mine ;-), to test it if you like.

@blurHY:

wasting resources

in what way?

blurHY · blurhy@zeroid.biton Aug 01, 2019Reply

@SSDifnskdjfnsdjk: The database is too big for ZeroNet, wasting resources. But you can try my IPZN when it's released

SSDifnskdjfnsdjk · ssdifnskdjfnsdjk@zeroid.biton Jul 31, 2019Reply

@SadLoli: yes, gitcenter is only git service on ZN so far. 500MB per database seems quite alot for a zeronet site, wondering what @blurhy: thinking about that, i think his horizon site had 300MB db 🙄 . Not everyone having SSD drive. PS: your site is loading ok, search not working yet - "Database query error: No results found for'loli'!".

Mg0 · mg0@zeroid.biton Jul 31, 2019Reply

@SadLoli: Hmm, I'd say that a thousand files is a good upper limit for starters, but I haven't done much extensive testing yet, I think the approach is to have small (below 2MB maybe?) content.json files, maybe even 10MB content.json files wouldn't be prohibitive given good hardware and network but that's just me wondering.

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@Kid Courageous: So GitCenter?

Kid Courageous · kidcourageous@zeroid.biton Jul 31, 2019Reply

@SadLoli go for it dude. I prefer gitrepo or gitlab over github

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

Here is what I have so far:
http://127.0.0.1:43110/19dfPUtRhw6KxzAYBzfntburwuZDQULMZ3

(Is it working?)

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@caryoscelus:

Hmm, i'm not sure what exactly is your purpose.

For now just building a site where one can search:

  • by SQLite Query
  • by metadataID
  • by multiple metadataIDs
  • by browsing through the data

So that I lean how do zeronet, basically and at some point basically this:

build a site on which users can upload whatever they have downloaded and then display it as it was intended

And looking at what @Sirlotaplot: said, that'll probably either be a fork of 1+Gallery or a collaboration with it since 1+Gallery is based of 0Gallery which itself was based of sadpanda/exhentai. Doesn't make much sense to reinvent the wheel here.

What is the shape of metadata you've got?

I'm not sure what you mean by 'shape'.
Metadata object in js after a query:
{
"title": String,
"pages": Integer,
"uploader": String,
"href": String,
"type": String,
"uploadDate": Integer,
"tags":[
{
"namespace":String,
"name":String,
"power":String
},
{...}
],
"SET_INFO": String,
"ID": String,
"PAGE": Inetger,
"IS_DATA_MISSING": Boolean
}

I'm currently signing what I have so far. And I've noticed that it takes a lot longer if I include a dbschema.json that will actually use the data to make a database. I think ZeroNet is rebuilding the database regardless of whether or not it needs to be rebuild on every signing.
The rebuild happens between:
[log]- css/index.css
and
[log]Adding timestamp and sha512sums to new content.json

caryoscelus · caryoscelus@zeroid.biton Jul 31, 2019Reply

@SadLoli:

That thing is difficult:

Some artists (both western and eastern) used it to upload their own works on there

Well, i guess that's enough to know ;)

The main purpose would be to search by id, which look like this: '/g/1330079/ea3aaac89f/'. Because most people have only downloaded the images, but not the metadata. Not sure what I can do there in terms of presorting

Hmm, i'm not sure what exactly is your purpose. Do you want to build a site on which users can upload whatever they have downloaded and then display it as it was intended? What is the shape of metadata you've got?

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@caryoscelus:

Nah, i meant what's the distribution terms / license? Were those author-uploaded pics, "illegal" copies, or just anything?

That thing is difficult:
Most stuff on there is commercial, but only really available in japan and in japanese. However some of the stuff was also from western artists.
Most artists didn't mind it being uploaded there, while some (historically) did. Some artists (both western and eastern) used it to upload their own works on there. Many didn't.

A lot of translators published their works(translations/edits like decensoring and many more) there. Some had their translations uploaded there of which some didn't want them to be there.

So in short: kind of?
Strictly answering the that question is inconclusive at best:
Were those author-uploaded pics, "illegal" copies?: No, except when they were.

You could think of sadpanda as a weaker enforced pixiv, if that helps.

AFAIK ZeroTalk and here are two most active places..

Good to know.

split by some thematic criteria and allow users to download only parts of db they're interested in

The main purpose would be to search by id, which look like this: '/g/1330079/ea3aaac89f/'. Because most people have only downloaded the images, but not the metadata. Not sure what I can do there in terms of presorting

caryoscelus · caryoscelus@zeroid.biton Jul 31, 2019Reply

@SadLoli:

I didn't mean compressed as in gzip/tar compressed, but compressed as in no indentation/spaces in the json file(with those it's about 1.3 or 1.8 GB in size).

Ah, that's usually called "minified" ;)

They are down/offline, ...

Nah, i meant what's the distribution terms / license? Were those author-uploaded pics, "illegal" copies, or just anything?

I've already seen that there is GitCenter, are there others on here?

It's possible to just host git repos as a zite, but i think the only hosting with interface is GitCenter

If you happen to find someone that does please point them in this direction. Millchan /l/ seems dead, is there somewhere else a more active community here on zeronet?

AFAIK ZeroTalk and here are two most active places..

should I make the json files optional or required? I don't think that zeronet can tell in which file the gallery metadata can be found in and querying the data only really makes sense if you have all of it to search through, right?

Since you're splitting anyway, i think it might be possible to split by some thematic criteria and allow users to download only parts of db they're interested in. Besides, sites with >10M required files have issues with public proxies. So i think it's still better to make them optional, though perhaps write a warning that search only works on loaded data (and ability to quickly load all of it (though there's that "distribute all optional files" button, many people may not know about it))

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@Kid Courageous: Sweet! Should I make a repo somewhere and how should be communicate?
I've already seen that there is GitCenter, are there others on here?

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@nekololi:

but would be happy to help seed as well as link any site that gets made in the process on loliguide.

Highly Appreciated!

loliguide

Nice!

i don't know offhand if there is anyone with zeronet development experience who is also interested in a hentai related project like this

If you happen to find someone that does please point them in this direction. Millchan /l/ seems dead, is there somewhere else a more active community here on zeronet?

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@Mg0: I guess proxies are a good poin't since they, if commercial servers, will have fast SSDs to quickly query the database.

just make sure to use content.json files properly. Not too many files for each

How many would you say that would be?

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@SSDifnskdjfnsdjk:

zeronet is not yet redy for large databases

The database worked fine for me(on SSD), it's ~530MB not sure if that's much.

Btw. i would like to help you, but as mentioned, i am not a developer nor a coder.

Dude, you are already helping a ton, as is everybody in this thread <3

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@Sirlotaplot:

blurhy's IPZN or something like it

Look interesting.

I'd bet it can handle 10x what I'm doing now (if it's linear), but that'd still only be 4TB.

That's good to know so we could split the ~50TB into like 13 Hubs and see from there. There's also the possibility that @1pgallery manages to improve efficiency so much that we don't need the hub thing, but I guess we just have to see for now.

Engine one has each board be it's own zite which means you only download the ones you want

@SSDifnskdjfnsdjk:

Millchan Engine is utilising merger sites - using "merger" feature of the zeronet.

Oh, okay thanks you two!

SadLoli · sadloli@zeroid.biton Jul 31, 2019Reply

@caryoscelus:

I don't think 0net can work with compressed JSON

I didn't mean compressed as in gzip/tar compressed, but compressed as in no indentation/spaces in the json file(with those it's about 1.3 or 1.8 GB in size).

Perhaps it might be a good idea to split it, though

Absolutely the full file took forever. to sign I've split it into about 3000 files with 600 entries each. It works reasonably well, database building takes ages however. But I'm not sure if I can do anything about that.

It is also possible to not use 0net-specific stuff and just parse metadata from JS, but that might be just too slow

I did the initial parsing in JS and I don't have enough fingers or toes to show you how many times the tab or even the browser crashed ; )

Most users will likely not want to seed all of the galleries

For now only the metadata. I think I'm gonna use that hub mechanic to combine it with the actual galleries.

I'm not familiar with the site

See @nekololi: s comment.

but what's their status?

They are down/offline, admins say that there is a full backup somewhere and the they'll look into more permanent storage options for the backup while telling people to get and save as much data to increase redundancy, so not sure how much the backup actually is a thing.

JS & 0net api (i still remember the pain of creating my meh engine..), but possibly i can help with some advice

Thanks, should I make the json files optional or required? I don't think that zeronet can tell in which file the gallery metadata can be found in and querying the data only really makes sense if you have all of it to search through, right?

Kid Courageous · kidcourageous@zeroid.biton Jul 30, 2019Reply

Happy to help. And I'm sorry for your loss.

nekololi · nekololi@zeroid.biton Jul 30, 2019Reply

@nekololi: just for clarity's sake, in case there is confusion from people who are not familiar with sadpanda, this issue isn't specifically about loli, although most lolicons I know are heavily invested in the site's demise. exhentai/e-hentai is a very old, well established site, and many hentai manga/doujin sites actually just republish content that gets published on e-hentai. many hentai lovers are affected by these recent events.

nekololi · nekololi@zeroid.biton Jul 30, 2019Reply

i have no zeronet development knowledge, but would be happy to help seed as well as link any site that gets made in the process on loliguide. unfortunately, i don't know offhand if there is anyone with zeronet development experience who is also interested in a hentai related project like this, but i wish you luck.

Mg0 · mg0@zeroid.biton Jul 30, 2019Reply

@SSDifnskdjfnsdjk: Hold it!

Rex research is working well since a while ago, I checked this proxy yesterday and it's actually synced with the zite (it's only downloaded about 2.5G out of 6.7G but I tried some links and it downloads on demand so it's working). The zite feels relatively light given its size, just make sure to use content.json files properly. Not too many files for each

SSDifnskdjfnsdjk · ssdifnskdjfnsdjk@zeroid.biton Jul 30, 2019Reply

@SadLoli: What's the difference between 'millchan' and 'millchan engine'? - go to Millchan, visit /dev board, then read first/pinned topic.. Millchan engine allow to spread optional files (images, videos) among multiple zeronet sites that are connected by the merging site (Millchan engine). Millchan Engine is utilising merger sites - using "merger" feature of the zeronet. You can sarch ZeroHello (Search in connected sites) or ZN docs or read this explanation about various zeronet features to get more details.
Note that it was talked earlier on zerotalk (or zerome?) that zeronet is not yet redy for large databases (hundreds of MB+ - @mg0: knows about that, he has large site here on ZN called Rex Research. I think good sites for alot of data is Millchan, Millchan Engine. Keep the db small, keep the required (not optional) file count small - is the key i think here on zeronet - at least until most of the people have fast and large NVMe drives and such. Btw. i would like to help you, but as mentioned, i am not a developer nor a coder.

Sirlotaplot · sirlotaplot@zeroid.biton Jul 30, 2019Reply

@SadLoli: IPFS is really interesting, I hope blurhy's IPZN or something like it takes off so we can use both together efficiently.

How do you reckon 1+Gallery would perform with ~50TB of shared data?

Hard to say. The viewer is dealing well with 5000 galleries. The hub manager was getting slow with 2000, but @1pgallery updated it recently and it's doing great now for my use case. I'd bet it can handle 10x what I'm doing now (if it's linear), but that'd still only be 4TB. I think it's also more to do with the number of files than the size of them.

The upside of a hub system is that you could always split the collection over multiple hubs so it doesn't matter that much whether the hubs can handle 1TB or 100TB. The part I'm more worried about is how well the gallery viewer will handle search requests.

The biggest advantage of ZeroGalleries over simple torrents is that they can be previewed, partially downloaded, liked and commented on.

Millchan and Millchan Engine

They are both made by the same dev, but the Engine one has each board be it's own zite which means you only download the ones you want. Unfortunately they are not compatible so boards are often doubled.

caryoscelus · caryoscelus@zeroid.biton Jul 30, 2019Reply

@SadLoli:

I have a ~530 MB compressed JSON file with all metadata for all sadpanda(with a few exceptions) galleries

Oh, wow, that's a lot.. I don't think 0net can work with compressed JSON, though. So if you want to utilize 0net db functionality, you'd need to store uncompressed ones (and possibly convert them, too). Perhaps it might be a good idea to split it, though. Actually, i've no idea what sort of metadata is in there, maybe it would be possible to cut out some of it?..

It is also possible to not use 0net-specific stuff and just parse metadata from JS, but that might be just too slow

What's the difference?

Most users will likely not want to seed all of the galleries ;) I thought you were going to host all of them on one site.. if, however, you want to have one index site and put individual galleries elsewhere, you may not need to bother with that.

I already really like that you can see the source of each website here.

There are people who are using code obfuscation, though..

You two seem to know quite a bit, would you or @caryoscelus: be interested in helping with development?

I'm not familiar with the site, so i'm not sure.. it's not like i care about content of drawn pictures, but what's their status? I'm not especially happy to help distributing copyrighted works, i think that helps copyright.

Barring that, i'm probably not excited about this enough to get heavily involved with JS & 0net api (i still remember the pain of creating my meh engine..), but possibly i can help with some advice

SadLoli · sadloli@zeroid.biton Jul 30, 2019Reply

@Sirlotaplot: Thanks! I'm happy to get this much response from here.

very thin silver lining to the fall of Sadpanda.

I guess, so. I'm actually looking forward to the discussion on p2p alternatives that will happen, once ATF gets back up. I guess that really is a plus. I've also been looking into IPFS recently, but it doesn't seem like you can have dynamic sites there, yet. Please correct me if I'm wrong, but I couldn't find anything in that regard.

I'm running two hentai manga focused ZeroGallery hubs

Sweet! Currently saving space for Exhentai backups, but looks interesting. How do you reckon 1+Gallery would perform with ~50TB of shared data?

>Millchan Engine

What's the difference between 'millchan' and 'millchan engine'?

Sirlotaplot · sirlotaplot@zeroid.biton Jul 30, 2019Reply

Welcome to ZeroNet! Happy to see some more interest in P2P networks from others in the hentai community, it's a very thin silver lining to the fall of Sadpanda.

I'm running two hentai manga focused ZeroGallery hubs: Alotaplot and Anthology.jp, totalling ~450GB (i.e. nothing), and I'm adding more every week. Short term there is absolutely no way to do this for the Sadpanda's archives due to their size, but I think that would be ideal in the long run. I also run a /h/entai board at Millchan Engine.

SadLoli · sadloli@zeroid.biton Jul 30, 2019Reply

@SSDifnskdjfnsdjk: Thanks, already digging through quite a bit of documentation, but more sources are always better.
Millchan is still loading, I think I'm gonna make a post there a bit later, when I actually have something to show.

i seen this static torrent index zite

I'll look into it.

I already really like that you can see the source of each website here.

You two seem to know quite a bit, would you or @caryoscelus: be interested in helping with development?

SSDifnskdjfnsdjk · ssdifnskdjfnsdjk@zeroid.biton Jul 30, 2019Reply

@SadLoli: yes, on Millchan are boards about loli, maybe you can find some people on it. I think some developers was also on Thunderwave zeronet site. --- You seem to want to develop not just static mirror site so you may need some d€veloper resources, find "devel" on ZeroNet Links site. Btw. isn't there user @nekololi: or @loligod: ?
PS: i seen this static torrent index zite having search/filtering feature, unsure if you can clone it and use some of its feature. Maybe i can help seed some files and i am not a developer, just know basic workings of ZN.

SadLoli · sadloli@zeroid.biton Jul 30, 2019Reply

@caryoscelus: I have a ~530 MB compressed JSON file with all metadata for all sadpanda(with a few exceptions) galleries. I want to make a searchable site by metadata ID(the locator part of the url to the gallery) and some browsing options, in the future at least.

Perhaps the only thing needed is to make sure all those big files are marked as optional

What's the difference?

As for the images I thought of registering the galleries by hash(probably sha1, because that's what BitTorrent uses) and giving out gallery IDs, which will be the xored hashes of all images of the gallery. This way the order in which the images are uploaded will not matter. The site will not allow duplicates to be uploaded checked by hash. Unless one offers a higher resolution.

It seems like scrapping data from servers is a bigger task

Scraping time is over, Sadpanda is dead.

caryoscelus · caryoscelus@zeroid.biton Jul 30, 2019Reply

I think for static (i.e. no user interaction) archive you don't need much 0net specific knowledge. If you have working copy on your disk, you can just put that into site directory. Perhaps the only thing needed is to make sure all those big files are marked as optional

It seems like scrapping data from servers is a bigger task, but that isn't related to 0net

That said, if you want users to be able to post parts of archive rather than putting everything by yourself, that's more complicated

SadLoli · sadloli@zeroid.biton Jul 30, 2019Reply

@Wormstweaker: Great! I think I'll post updates here on this ZeroMe account.

Wormstweaker · wormstweaker@zeroid.biton Jul 30, 2019Reply

@SadLoli: Welp, I'll try to spread the word, and my zeronet instance should be running a lot, so I guess I can help seed?... ¯_(ツ)_/¯

SadLoli · sadloli@zeroid.biton Jul 30, 2019Reply

@SadLoli: Why are your mentions of me linked by my mentions of you aren't?

SadLoli · sadloli@zeroid.biton Jul 30, 2019Reply

@Wormstweaker: You don't necessarily need to have knowledge of zeronet to help. Spreading word around would also be of great value especially here, where we need seeders to remain online. Getting more people also increases the chances that someone with actual skill might choose to help.

Oh and crap, this is more of an image board

I've been around, I actually prefer imageboards to reddit.

Wormstweaker · wormstweaker@zeroid.biton Jul 30, 2019Reply

@SadLoli: Oh and crap, this is more of an image board than a reddit/4chan like
Still, try on zites that look like these, you might get the support you need

Wormstweaker · wormstweaker@zeroid.biton Jul 30, 2019Reply

@SadLoli: I'm interested of course, but I have little to no knowledge of zeronet
So I can't really help, but feel free to keep me updated on your progress!

SadLoli · sadloli@zeroid.biton Jul 30, 2019Reply

@Wormstweaker: Sweet! Gonna give it a try.
Are you by any chance interested in helping?

Wormstweaker · wormstweaker@zeroid.biton Jul 30, 2019Reply

@SadLoli: Hello! Maybe you could try to create a thread on there, some people might help you with your project
(This is MillChan btw, a reddit like zite)
http://127.0.0.1/1ADQAHsqsie5PBeQhQgjcKmUu3qdPFg6aA

This page is a snapshot of ZeroNet. Start your own ZeroNet for complete experience. Learn More