Infrastructure for binary part of our project

Anything related to dev. & that doesn't fit in below categories.
User avatar
Q_x
developer
Posts:334
Joined:Thu Sep 23, 2010 15:10
Infrastructure for binary part of our project

Post by Q_x » Sun Jun 12, 2011 17:17

There is a problem I'm facing every time when uploading illustrations.
Problem is with binary files - slow to transfer, and they produce bloat. So it's not only slow and bloaty, but also slower and bloated more with time. Those are especially painful thing when using lightweight checkout as a local copy, with bazaar branch the speed is tolerable (for now), and the size is close to 1 GB, which is still not that bad. But it will be.
Also, version control is not designed to handle repositories that big, which consist of like 2% of text and 98% of bins.

Possible solutions:
dropbox, e-storage of any kind - it costs real money
ftp (-like) server - possibly free, easy, but needs some extra maintenance and planning to prevent vandalism and data loss.
using cvs that actually deals with binary files well, even if it's something really old and strange.
nothing else comes to my mind

Problems:
Providing artists with easy to use upload infrastructure
Providing devs with the same as above, plus some specs on how to use the files - like how to provide info on authorship, if it is better to link or embed the images in Inkscape, or how to (re-)structure directories when working with scribus files.

Also the problem is how to separate the content into what needs CVS and what not - for example I think Scribus files and Inkscape files vital for the project (templates, logos) could be versioned, even without linked assets.
I'm the filthy bastard you wish you never met.
User avatar
snowdrop
developer
Posts:798
Joined:Mon Feb 01, 2010 15:25
Location:Sweden
Contact:

Re: Infrastructure for binary part of our project

Post by snowdrop » Tue Jun 14, 2011 07:42

For simplicity I think dropbox-like services are the best option. It doesn't matter if they cost real money, I'll pay - they're not that expensive.

Here are some factoids;

dropbox

Supports 30-day or unlimited "version control":
Dropbox keeps snapshots of every saved change in your Dropbox folder over the last 30 days (or more with the Pack-Rat feature). So if your pet accidentally pressed the delete key and erased your memoirs or you simply saved a bad change, you can restore the file with a few clicks./../

If recovering files becomes a recurring issue, you can upgrade your account include the Pack-Rat feature. Doing so gives you unlimited snapshots of your files, allowing you to go recover any file as far back in time as you like
See https://www.dropbox.com/help/11 for more info.

Prices: 99 USD/year for 50 GB, or 138 USD/year for 50 GB and eternal version control ("pack rat"). Meaning it costs 2,76 per GB with eternal version control or 1,98 with 30-day limited version control.

There seems to exist clients for all major platforms, but they're not even needed to upload/download something.

ubuntu one
Seems to lack all kinds of version handling, but this is not confirmed yet.

Price: 30 USD/year per 20 GB, giving us a cost of 1,5 per GB stored data.

Clients exist for Linux & Win.
User avatar
Q_x
developer
Posts:334
Joined:Thu Sep 23, 2010 15:10

Re: Infrastructure for binary part of our project

Post by Q_x » Tue Jun 14, 2011 10:01

I'd rather avoid all the stuff tagged with U, even if it's opensource.
they give 2GB there, it's not bad, just same as dropbox, also with browser-only mode.

Also, we can increase the storage on our dropbox account slightly, registerring accounts with this link:
http://db.tt/UjBPzMm
I'm the filthy bastard you wish you never met.
User avatar
snowdrop
developer
Posts:798
Joined:Mon Feb 01, 2010 15:25
Location:Sweden
Contact:

Re: Infrastructure for binary part of our project

Post by snowdrop » Wed Jun 15, 2011 22:14

User avatar
Q_x
developer
Posts:334
Joined:Thu Sep 23, 2010 15:10

Re: Infrastructure for binary part of our project

Post by Q_x » Thu Jun 16, 2011 09:09

Thumbs up for using 0.1 alpha! Just... don't count me in. :P :P :P :lol:
I'm the filthy bastard you wish you never met.
User avatar
Q_x
developer
Posts:334
Joined:Thu Sep 23, 2010 15:10

Re: Infrastructure for binary part of our project

Post by Q_x » Thu Jun 16, 2011 12:38

AFTER checking out how expensive the hosting is, I think we can stick to the solution partially similar to what we have already - that is keeping incoming stuff on free dropbox account and storing/syncing, and (S-) FTP-ing bloat in a central place with lower availability (this may be even done in complete anarchy, like when syncing in direct p2p manner with p300 - which is not a good piece of software BTW, rsync or whatever with backup kept locally by everyone involved).

Also:
http://stackoverflow.com/questions/1044 ... r-binaries
This
http://code.google.com/p/boar/
is still in beta
I'm the filthy bastard you wish you never met.
User avatar
snowdrop
developer
Posts:798
Joined:Mon Feb 01, 2010 15:25
Location:Sweden
Contact:

Re: Infrastructure for binary part of our project

Post by snowdrop » Mon Jun 20, 2011 09:07

I don't get it: How exactly does Board help us sync between us? From what I understood it's purely local...or am I missing out on something?

Sparkleshare has made plenty progress and they are contemplating going the same road as syncany with introducing hosting on FTP etc. The two projects seem to also be aware of each other, which is a good sign. :P I don't think we should use Sparkle unless it has other storage options than VCS:s.

I'm currently trying to get an understanding of how (un)stable syncany is, as I still thiknk it's the best - at current - stuff out there "on paper". Maybe should just run it in private on my machine for a while and see how it performs
User avatar
Q_x
developer
Posts:334
Joined:Thu Sep 23, 2010 15:10

Re: Infrastructure for binary part of our project

Post by Q_x » Tue Jun 21, 2011 10:21

Sir, you are just about to check alpha software, three months old. Doing this in the thread related to us looking for storage solution makes me think you will propose it to be used for production purposes, which I said already about.

In my everyday life, that is hobby and work, I struggle with software that needs more care than it got. I lose time, energy and nerves doing what is normally well-paid work, that is testing software and reporting bugs that are later simply ignored, or were "fixed" already in the way that does not fix anything (like when the text is supposed to wrap on the edge of a line, but it doesn't wrap, and the solution to that bug, that marks the bug as fixed, is putting manual line breaks... Sad but true reality.). There are some rare occasions when this is after all good - that is the case of using bleeding edge Scribus version, which gets lots of bugfixes and features we need, and not much stability loss (if any). But this is post-alpha software.

Now, I'm not saying that what you consider us to use is bad software, or bad developers. It's quite opposite: I understand what alpha means and I think devs are responsible, tagging the software with alpha.

Boar is VCS for bin files, not only local, but remote as well.

I'm not going to waste time testing things now. I (or "we", whoever that will be) will have to evaluate everything as soon as it will be at least in "usable beta", no matter how test would go now. For now the only right policy for me is to use software as it is, without relying on devs that may, or may not, push things forward in the future.
I'm the filthy bastard you wish you never met.
aspidites
Posts:101
Joined:Mon Apr 11, 2011 22:39

Re: Infrastructure for binary part of our project

Post by aspidites » Wed Jun 22, 2011 15:04

Q_x, you say that you aren't going to waste time testing things now, and that you are waiting for usable betas. For syncany this makes sense, but boar's website says this:
The project is in beta phase. There is no fancy GUI - only a command line client. It is very much usable, but not very polished.
That sounds like 'usable beta' to me. I think you even pointed out the project. So you are testing this now, right? :-)
User avatar
snowdrop
developer
Posts:798
Joined:Mon Feb 01, 2010 15:25
Location:Sweden
Contact:

Re: Infrastructure for binary part of our project

Post by snowdrop » Fri Jun 24, 2011 20:53

I agree that it's bad to rely on developers creating this and that solution that is supposedly a milestone and coming feature in a project, especially in open source projects where continuity and spent work hours aren't guaranteed.

Even if it is so we still are always in the mercy of the coders and it's also decided we prefer to use floss instead of commercial software if possible and not creating too much work flow problemas.

Yes, there is an immense difference between stable and proven projects that have been around and up and coming projects like Sparkleshare. While I personally believe that Sparkle and/or Syncany could become huge and well developed projects, it might take very long time for that to happen and for the code to be usable in a production environment. I'm aware of that, and it is of course nor reasonable for us to wait for e.g. 2 years for that to happen.

I do however not think it's very interesting how different dev crews describe the state of their software. If it is called alfa, beta or RC is the same to me - what I'm interested in is if it fill the correct functions for us and if it works as expected. Alfa x could be way crappier than Beta Y, while released Z could be shittiest of them all. In short, I think testing stuff is ok and more important than the labels, that, and general feedback from others ofc...
Post Reply