SciLINC Update

Message boards : Cafe : SciLINC Update

Reply to this thread
Subscribe to this thread
Sort
AuthorMessage
Ron Parker
Forum moderator
Project administrator
Project developer
Avatar
Joined: Mar 27, 2007
Posts: 14
ID: 1
Credit: 1,232
RAC: 13
Message 42 - Posted 22 Jun 2007 15:39:55 UTC
Last modified: 22 Jun 2007 15:56:20 UTC

When development of the SciLINC project began it had four primary goals. Edited for brevity, they were:

1. Increase public access to nationally significant scientific literature.

2. Enhance the usefulness of digitized materials by creating a Web repository of scanned literature, keywords, and online resources with tools for searching and analysis.

3. Create an educational tool for learning about plant life. While the screensaver application is indexing keywords, the participant's computer will display information about plant life within the United States and around the world. The information displayed will describe each plant name or term currently being indexed on the participant's computer, and will include descriptive data, images, maps, and the annotated outlinks for that term.

4. Provide a model for adopting public-resource computing applications within the library community.


Botanicus is doing a wonderful job of meeting goals 1 and 2 including processing data generated by SciLINC. The project has certainly also meet goal 4.

We have learned much about grid-based, distributed, public-resource computing applications and the BOINC architecture. There are thoughts and plans for analyses down the road that will be much more computationally intensive than the original SciLINC analysis and we look forward in time to bringing these projects to you.

While the amount of data that SciLINC has to analyze will increase greatly in the days ahead it does not appear that increasing the volume of information is going to improve the user experience of running the SciLINC client.

It has been suggested that we repackage our data into single files instead of uploading and downloading 50 files per workunit as we currently do. This suggestion has been heeded and implemented. We had planned on doing it before SciLINC was rolled out but scheduling prevented it and the community discovered the project before we were ready to announce it. We expect that testing will show the repackaging lessens the load placed upon the core BOINC client software. But, it does not change the amount of data being transferred.

The truth is that the workunits fly by so rapidly that implementing goal 3 never became realistic.

When development of SciLINC began, the project lead's understanding was that from a technological and economic standpoint it makes sense to use public-resource computing in place of an internal grid computing architecture whenever less than a gigabyte of data is required per cpu-day of computation. Using the BOINC framework to transfer the data to clients, SciLINC meets this volume-of-computation guideline.

However, our brief experience with the dedicated BOINC community over the last couple weeks has shown that, to the community these numbers may differ somewhat. In its original form SciLINC would have needed to transfer roughly 250MiB of compressed data in order to occupy a modern CPU for a day. This would expand to nearly 660MiB of input data. Then the client would need to upload about 44MiB of results which would compress to 17MiB. These numbers have only grown as SciLINC has been improved and made more efficient.

This is not acceptable to the average BOINC user.

Looking at the numbers from the perspective of someone on dial-up, if they set SciLINC to only 1% of their BOINC time, this would be roughly 15 minutes out of a day. For this 15 minutes they would have needed to download around 2.5MiB of data. This may not be a huge issue for broadband users, but if someone is on dial-up (as we have learned many BOINC fans still are) the transfer time would exceed the computation time.

So, where are we now?

Even if the transfer:credit ratios were acceptable to the community, we do not have enough data to realistically occupy hundred or thousands of BOINC enthusiasts for a lengthy period of time. As we have already seen on various community boards a relatively small amount of credit is earned for a comparatively large load on their system resources. Any computational and transport related improvements that have been tested have only resulted in more data needing to be transferred.

As stated above, we are investigating the possibility of performing much more computationally intensive analyses in the months ahead. It is expected that these will be a much better fit for a BOINC project than the current task of text-indexing and taxonomic analysis which has a relatively low mathematical complexity.

Because of this it has been decided that for now all SciLINC computation will be performed internally. When we have something with a better credit-reward ratio (and nicer screensaver) it will be made available to the community.

Thank you again for your interest and support. We look forward to working with you in the future.

The SciLINC Team

Shai Hulud
Joined: Jun 11, 2007
Posts: 5
ID: 38
Credit: 1
RAC: 0
Message 43 - Posted 22 Jun 2007 19:12:19 UTC
Last modified: 22 Jun 2007 19:27:49 UTC

Hi,

I'm one of those dial-up-users and imho, you dont need to take dial-up users into your consideration. More precisely, dialup-users are no reason not to continue this project.

There have always been and there will always be projects which are not suitable for EVERY BOINC-User.

PS3GRID is for PS3-Owners only, there are Projects that dont support either Linux or Windows, there are quite a few other projects with high uploads/downloads (SIMAP for example is a no-go for dial-uppers actually, but still I support them)...

Even as a dial-upper, if I find a project interesting I would download a huge workunit - in the meantime its crunching for a different project anyway! So no cpu-time lost!

Besides, offering wu's which take appr. 1 to 2 hours are enough anyway... More than enough time for the screensaver, imho. (Would make 25MB to download and 1.7MB to upload - normal business even for me as a dial-upper!)

Furthermore, many people install BOINC as a service, so screensavers arent available anyway!

Even if BOINC is installed normally, many people disabled the screensaver, cause it could take away some ressources, especially on slower computers... I watch it very rarely! And in the end: Its a screensaver, not a video film I will watch for 2 hours...

Well, so even very short wu's wont be a problem. Chess960-WU's are taking less than 3 minutes on my puter... There are a few projects with even shorter wu's!

And if we'd finish the wu's in a short time, there are other projects as well that have work very seldom (Pirates for example)... I'm a patient man...

Well, to make a long story short: Please continue SciLINC. As I wrote in my profile and mentioned on my team sites forums, this is truly a VERY UNIQUE Project I would love to support! The earlier the better!

Only problem was the numerous files we needed to upload! If you fixed this problem! Great! Keep those wu's coming!

Best regards, Steffen

PS: But in the end, as long as you come back some time in the future, I'm fine with this also!
____________


My NEW BOINC-Site

Why people joined BOINC Synergy...

=Lupus=
Joined: Jun 11, 2007
Posts: 1
ID: 47
Credit: 1
RAC: 0
Message 44 - Posted 22 Jun 2007 23:01:33 UTC
Last modified: 22 Jun 2007 23:03:13 UTC

My opinion is:

If you can give us work after implementing the single-file-solution, DO IT!

We are happy to give you our computing power, and because most of us share this project with some others, it doesn't bother us if the transfer is above average - I know of an other project that downloads and uploads 700k for 15 minutes of work, means 130MB every day! And there is noone ever whining about that...

I too think this is a very unique project that is worth supporting, and hope that you continue your work with "public access"! (I mean not-"only-internal", english is not my native language)

Julian Ellis
Joined: Jun 11, 2007
Posts: 2
ID: 65
Credit: 0
RAC: 0
Message 45 - Posted 22 Jun 2007 23:39:57 UTC

I totally agree, once you sort out the problem of uploads/downloads there is no reason to stop working with Boinc.

I personally run over 40 different projects and many of them use large data files which only run for a short time and as previously stated some projects very rarely have work (I have waited months sometimes to get a pirates WU and then sometimes only get 1 or 2 and they have run out again).

Some Boinc users may find it unsuitable for their slow computers or dial up connections and others may just choose not to run it because it does not suit the way they want to do work, but that will just mean more work will be availiable for those of us that choose to stick at it because we want to help with a project that we think is worthwhile.
____________

WTBroughton
Joined: Jun 12, 2007
Posts: 1
ID: 146
Credit: 1
RAC: 0
Message 46 - Posted 23 Jun 2007 7:54:12 UTC

I agree with the comments so far. I also run 40+ BOINC projects, but when SciLINC started I gave it 80% of my resources (3 of my top CPU's). This is a very interesting project with a lot of potential, I would hate to see it disappear especially as you Ron have done a first class job so far.
I will stay linked and look forward to future crunching.

Best Regards
Tom Broughton

Dick Duggan
Joined: Jun 14, 2007
Posts: 1
ID: 205
Credit: 4
RAC: 0
Message 48 - Posted 23 Jun 2007 9:57:52 UTC

One more post with the same message.

As was said above, not every BOINC project is suitable for every BOINC user. I'm another 40+ project person but there are a couple I choose to run only periodically because I don't like their resource usage. I'm a big boy, I can figure that out. :-)

Hope to see more wu's.

Trog Dog
Avatar
Joined: Jun 11, 2007
Posts: 2
ID: 113
Credit: 0
RAC: 0
Message 49 - Posted 24 Jun 2007 2:06:08 UTC

The only problem that I had with this project was that DOS'd each one of my hosts. If that problem has been fixed then let the wu's flow.

If you feel that the data transfers will be a source of complaint then highlight this on the front page and let us crunchers decide. As far as the credits go there was discussion at the last BOINCfest about awarding credits for data transferred - I don't know how far this has progressed, but if you feel that is also a potential source of complaint, then it's another avenue to explore.

____________
Free Tibet/ Tibet Libre Team Forum

caspr
Avatar
Joined: Jun 16, 2007
Posts: 9
ID: 283
Credit: 0
RAC: 0
Message 50 - Posted 24 Jun 2007 8:55:37 UTC

As stated above, we are investigating the possibility of performing much more computationally intensive analyses in the months ahead. It is expected that these will be a much better fit for a BOINC project than the current task of text-indexing and taxonomic analysis which has a relatively low mathematical complexity.

Because of this it has been decided that for now all SciLINC computation will be performed internally. When we have something with a better credit-reward ratio (and nicer screensaver) it will be made available to the community.



Hey Ron, Let us help with whatever!! I think we all pretty much know what we're getting into. Personally, I don't use the screensaver feature...eats up prosscesser power! You never know, we might even have an idea or 2 to help!

cfreeland
Forum moderator
Project administrator
Joined: Apr 2, 2007
Posts: 5
ID: 2
Credit: 780
RAC: 34
Message 54 - Posted 27 Jun 2007 14:55:06 UTC

This is Chris Freeland, Project Manager for SciLINC, and I'm really encouraged by the "Save SciLINC" camapaign that's happening here and on other boards. Because of your feedback and suggestions we've found ways to keep SciLINC running and distributing work units to anyone who wants to help, so expect to see some work units by the weekend.

As Ron posted we want to keep doing more projects through BOINC and have begun some preliminary work on using BOINC for phylogenetic analysis. We'll be sure to keep you posted on progress.

Thanks again for all your help and please keep the suggestions coming!

Chris



Barbud [USA]
Avatar
Joined: Jun 7, 2007
Posts: 4
ID: 5
Credit: 2
RAC: 0
Message 55 - Posted 27 Jun 2007 21:55:07 UTC - in response to Message ID 54.

Because of your feedback and suggestions we've found ways to keep SciLINC running and distributing work units to anyone who wants to help, so expect to see some work units by the weekend.


Look forward to testing out the new wu's.

As Ron posted we want to keep doing more projects through BOINC and have begun some preliminary work on using BOINC for phylogenetic analysis. We'll be sure to keep you posted on progress.


I'll leave this quote for one of the ATA regulars.

Bring on the banana's

____________

Shai Hulud
Joined: Jun 11, 2007
Posts: 5
ID: 38
Credit: 1
RAC: 0
Message 56 - Posted 28 Jun 2007 21:06:14 UTC - in response to Message ID 54.

This is Chris Freeland, Project Manager for SciLINC, and I'm really encouraged by the "Save SciLINC" camapaign that's happening here and on other boards. Because of your feedback and suggestions we've found ways to keep SciLINC running and distributing work units to anyone who wants to help, so expect to see some work units by the weekend.

As Ron posted we want to keep doing more projects through BOINC and have begun some preliminary work on using BOINC for phylogenetic analysis. We'll be sure to keep you posted on progress.

Thanks again for all your help and please keep the suggestions coming!

Chris


Thats really great news! Cant wait to support your project more!

Thanx!

____________


My NEW BOINC-Site

Why people joined BOINC Synergy...

zombie67
Avatar
Joined: Jun 15, 2007
Posts: 7
ID: 228
Credit: 0
RAC: 0
Message 60 - Posted 7 Jul 2007 15:41:49 UTC - in response to Message ID 54.

This is Chris Freeland, Project Manager for SciLINC, and I'm really encouraged by the "Save SciLINC" camapaign that's happening here and on other boards. Because of your feedback and suggestions we've found ways to keep SciLINC running and distributing work units to anyone who wants to help, so expect to see some work units by the weekend.


Any news?
____________
Dublin, CA
SETI.USA
Stats
My stuff
BOINC IRC chat

Shai Hulud
Joined: Jun 11, 2007
Posts: 5
ID: 38
Credit: 1
RAC: 0
Message 67 - Posted 16 Jul 2007 18:42:17 UTC - in response to Message ID 60.

This is Chris Freeland, Project Manager for SciLINC, and I'm really encouraged by the "Save SciLINC" camapaign that's happening here and on other boards. Because of your feedback and suggestions we've found ways to keep SciLINC running and distributing work units to anyone who wants to help, so expect to see some work units by the weekend.


Any news?


Same question here! Isnt it time for an update, please?
____________


My NEW BOINC-Site

Why people joined BOINC Synergy...

cfreeland
Forum moderator
Project administrator
Joined: Apr 2, 2007
Posts: 5
ID: 2
Credit: 780
RAC: 34
Message 69 - Posted 17 Jul 2007 12:38:20 UTC

Sorry folks, I wasn't receiving update messages for this thread. We'll be running again by the end of this week.

Shai Hulud
Joined: Jun 11, 2007
Posts: 5
ID: 38
Credit: 1
RAC: 0
Message 70 - Posted 17 Jul 2007 16:51:22 UTC - in response to Message ID 69.

Sorry folks, I wasn't receiving update messages for this thread. We'll be running again by the end of this week.


Hey, great news! Thanx! :)
____________


My NEW BOINC-Site

Why people joined BOINC Synergy...

zombie67
Avatar
Joined: Jun 15, 2007
Posts: 7
ID: 228
Credit: 0
RAC: 0
Message 74 - Posted 26 Jul 2007 5:14:19 UTC - in response to Message ID 69.

Sorry folks, I wasn't receiving update messages for this thread. We'll be running again by the end of this week.


Any news?
____________
Dublin, CA
SETI.USA
Stats
My stuff
BOINC IRC chat

HomeGnome
Joined: Jun 12, 2007
Posts: 3
ID: 151
Credit: 0
RAC: 0
Message 75 - Posted 31 Jul 2007 11:03:17 UTC - in response to Message ID 69.

Sorry folks, I wasn't receiving update messages for this thread. We'll be running again by the end of this week.


This week or the next week?

[AF>Futura Sciences>Linux] Thrr-Gilag
Avatar
Joined: Jun 11, 2007
Posts: 3
ID: 52
Credit: 2
RAC: 0
Message 76 - Posted 3 Aug 2007 12:10:40 UTC

This one or next. Just have to wait ^^
____________
In my fiction I am careful to make everything probable and to tie up all loose ends. Real life is not hampered by such considerations. (Isaac Asimov)

cfreeland
Forum moderator
Project administrator
Joined: Apr 2, 2007
Posts: 5
ID: 2
Credit: 780
RAC: 34
Message 77 - Posted 3 Aug 2007 12:20:56 UTC

Seems we're running into issues getting work queued. We've resolved all the other issues, just this one hanging us up. We'll post here and send out an announcement when we're ready for you to resume running SciLINC. Thanks for your continued interest.

caspr
Avatar
Joined: Jun 16, 2007
Posts: 9
ID: 283
Credit: 0
RAC: 0
Message 83 - Posted 8 Sep 2007 5:37:42 UTC

sooooo, I guess this project is dead? the issues won? Or are there still gonna be wu's by the weekend? :o)

[AF>Futura Sciences>Linux] Thrr-Gilag
Avatar
Joined: Jun 11, 2007
Posts: 3
ID: 52
Credit: 2
RAC: 0
Message 84 - Posted 12 Sep 2007 23:44:18 UTC

Knok knok.
____________
In my fiction I am careful to make everything probable and to tie up all loose ends. Real life is not hampered by such considerations. (Isaac Asimov)

cfreeland
Forum moderator
Project administrator
Joined: Apr 2, 2007
Posts: 5
ID: 2
Credit: 780
RAC: 34
Message 88 - Posted 18 Sep 2007 12:58:59 UTC
Last modified: 18 Sep 2007 13:01:01 UTC

Who's there? Despite what you might think, we are. Long story short, the project is not dead, but may change framework to successfully deliver our intended outcomes. As previously mentioned, this summer we ran internal testing and came up with some fundamental bugs with our taxonomic name matching algorithms. As we worked through those problems, we then had hardware malfunctions. Then, our main developer moved on to another gig. You know, typical problems faced by every development team.

In the mean time, we continued working with the folks at uBio.org, who have also developed taxonomic name matching algorithms (under an umbrella of what they call 'taxonomic intelligence'). Their algorthims are service-based and performant, and initial evaluation has suggested that they are also more accurate than the code we co-opted for SciLINC. Given that, and that text indexing is not a processor intensive exercise (something we knew going into the project), we're at a decision point of whether to continue this project using BOINC, which may ultimately prove an unsuccessful implementation of the technology for the previous reasons, or to switch to a SOA using uBio.

To be honest, we're torn. We set out to demonstrate BOINC within scientific literature and to build a community of enthusiasts around the subject matter, but the problems listed above have given us pause on direction. I'd be interested to hear your (collective) thoughts.

caspr
Avatar
Joined: Jun 16, 2007
Posts: 9
ID: 283
Credit: 0
RAC: 0
Message 89 - Posted 19 Sep 2007 15:40:02 UTC


To be honest, we're torn. We set out to demonstrate BOINC within scientific literature and to build a community of enthusiasts around the subject matter, but the problems listed above have given us pause on direction. I'd be interested to hear your (collective) thoughts








Of course the final decision is your's but personally I'd really like to work on the project. And seeing as I am completely computer illiterate,I can't help with the BOINC side of it. But you may be surprised at all the help these folks can provide,they're a pretty good group!

Trog Dog
Avatar
Joined: Jun 11, 2007
Posts: 2
ID: 113
Credit: 0
RAC: 0
Message 90 - Posted 22 Sep 2007 8:42:31 UTC - in response to Message ID 89.


To be honest, we're torn. We set out to demonstrate BOINC within scientific literature and to build a community of enthusiasts around the subject matter, but the problems listed above have given us pause on direction. I'd be interested to hear your (collective) thoughts




My previous comments still stand. If you can deliver wu's that won't kill my machines then go for it.
____________
Free Tibet/ Tibet Libre Team Forum

Keck_Komputers
Avatar
Joined: Jun 12, 2007
Posts: 2
ID: 140
Credit: 0
RAC: 0
Message 91 - Posted 22 Sep 2007 10:05:28 UTC

I still have a host polling this project. I do hope it isn't DOS'ed if this project comes back to life.

One thing within the BOINC framework that can help prevent the previous DOS attacks is updating to the latest server code and setting the max_wus_in_progress tag to prevent masses of concurrent file transfers.
____________
BOINC WIKI

BOINCing since 2002/12/8

[B^S] DISCOMgsvr
Joined: Jun 11, 2007
Posts: 1
ID: 67
Credit: 0
RAC: 0
Message 92 - Posted 22 Sep 2007 13:16:06 UTC

SciLINC could continue as a BOINC Project.

There doesnt have to be work all the time.

[SETI.USA]Tank_Master
Joined: Jun 10, 2007
Posts: 1
ID: 34
Credit: 0
RAC: 0
Message 93 - Posted 22 Sep 2007 14:50:06 UTC

I to hope that you can continue with the BOINC development. The latest client build (5.10.20) has done some work in reducing the impact on the host with several small files wating to u/l or d/l. There will also probably be further development in this area in the new 6.1.x versions. If you have any suggestions on changes to the server side code, now would be a great time to work with the devopers of BOINC, for they are in the process if changing a lot of the code as it is.

Augustine
Joined: Jun 11, 2007
Posts: 3
ID: 98
Credit: 2
RAC: 0
Message 94 - Posted 22 Sep 2007 15:30:19 UTC

If you're having second thoughts about using BOINC because the new algorithm won't require that much CPU, I'd say that there have been a few non CPU-intensive BOINC projects before, with at least one still alive, DepSpid.

As a matter of fact, for reasons that are perhaps obvious, only non CPU-intensive projects are run on my laptops. Therefore, I'd be a sure contributor to this project is if fits this profile.

However, as you've lost a developer and using the new algorithm also means using an application that already works, I can where you're torn apart. For then porting the new algorithm to BOINC would require man-hours you cannot afford.

I would suggest though that you look at wrapping the new algorithm in BOINC. "Wrapping" is a technique available in BOINC in which any application is controlled by a BOINC-compliant program, thereby eliminating the need to do a major port of an application to BOINC.

Of course, at the end of the day, it's your decision between achieving your results with or without BOINC. On my part, I'd be glad to contribute to this effort with BOINC.
____________

Nightbird
Joined: Jun 11, 2007
Posts: 2
ID: 40
Credit: 0
RAC: 0
Message 95 - Posted 4 Oct 2007 19:15:47 UTC

Hello,
Try to go on with Boinc.
Our experience with you and your experience with us will be useful.
Try the Boinc wrapper if easier for you, contact Berkeley if needeed.
We are waiting for you. ;)

Nightbird
Joined: Jun 11, 2007
Posts: 2
ID: 40
Credit: 0
RAC: 0
Message 96 - Posted 29 Oct 2007 20:18:47 UTC

Up ;)
Guess what's the question...

Any news ?

ThEfT
Joined: Jun 10, 2007
Posts: 1
ID: 19
Credit: 1
RAC: 0
Message 98 - Posted 31 Oct 2007 20:36:06 UTC - in response to Message ID 88.
Last modified: 31 Oct 2007 20:36:28 UTC

Who's there? Despite what you might think, we are. Long story short, the project is not dead, but may change framework to successfully deliver our intended outcomes. As previously mentioned, this summer we ran internal testing and came up with some fundamental bugs with our taxonomic name matching algorithms. As we worked through those problems, we then had hardware malfunctions. Then, our main developer moved on to another gig. You know, typical problems faced by every development team.

In the mean time, we continued working with the folks at uBio.org, who have also developed taxonomic name matching algorithms (under an umbrella of what they call 'taxonomic intelligence'). Their algorthims are service-based and performant, and initial evaluation has suggested that they are also more accurate than the code we co-opted for SciLINC. Given that, and that text indexing is not a processor intensive exercise (something we knew going into the project), we're at a decision point of whether to continue this project using BOINC, which may ultimately prove an unsuccessful implementation of the technology for the previous reasons, or to switch to a SOA using uBio.

To be honest, we're torn. We set out to demonstrate BOINC within scientific literature and to build a community of enthusiasts around the subject matter, but the problems listed above have given us pause on direction. I'd be interested to hear your (collective) thoughts.

XtremLab seems now to switch to being non cpu intensive, too. So this should not matter. I think it's worth the effort to use BOINC because you'll find a real big community with scientific interests. And you may probably also find some development help ;)

[AF>Linux>Fight] xipehuz
Joined: Jun 11, 2007
Posts: 1
ID: 86
Credit: 0
RAC: 0
Message 100 - Posted 23 Nov 2007 16:47:14 UTC

For me, it all boils down to this :

What is your main objective in the long run ?

Building a community of enthusiasts or delivering the goods to your "clients".

If it's the former, then definitely go for Boinc. We are a huge community and we are all science and technology enthusiasts.

If it's the latter, then go for uBio's solution. It'll save you time and money and frustration of working with a bunch of geeky enthusiasts.

My 2 cents.

Reply to this thread

Message boards : Cafe : SciLINC Update



Return to SciLINC main page


Copyright © 2008-2021 Missouri Botanical Garden
Written by Ron Parker