Jump to content


Photo

Extend Utils


  • Please log in to reply
65 replies to this topic

#16 Stuckie

Stuckie

    GP Mania

  • GP32 Hardcore
  • PipPipPipPipPip
  • 432 posts
  • Gender:Male
  • Location:Tired, and sleeping in the basement...

Posted 28 June 2011 - 08:03 AM

Check mount to ensure that it's put the home overlay on properly, and post up the script file it spat out. ( or post up both if you're not sure what you're looking for in the mount spam. )
I would also manually check /home to see if the files you places on the overlay are there as well - as that'll be a sign of whether it's behaved itself or not.

I'll try find some time to rework my guide to include mounting a home overlay tonight, if I get in early enough.

-edit-
Actually, have some time to do it just now....

In the case of the mount info, you're looking for a line reading "aufs on /home type aufs ....."

Anyway, from the beginning!
Download Extend Utils and stick it somewhere in your pandora directory structure: http://stuckiegamez....ExtendUtils.pnd
Run it.
Select "New Extend/Overlay/Swap"
Select "New Overlay"
Select a size ( for my purposes, I'm just testing so 256mb ) .. if you select custom, prepare for a long wait.
Place the file on your SD card and give it a name, we shall call it "homeMount"... you don't need to add .overlay or .extend, as it does it for you.
It'll now either extract a premade empty file, or it'll DD a new file, taking forever, and probably raping your SD in the process.. you have been warned ;)
You'll be pushed back to the main menu now.

Select "New Pre-Mount"
Select "New Overlay Mount Script"
Select "Standard Home Overlay"
Choose your file that you saved on card in the previous stage.
Save the mount script anywhere you like, calling it anything you like.. for our purposes, on the SD card named "homeMount"
Back to the main menu.

Select "Mount Existing Pre-Mount"
Gotcha: As we've saved the script to a FAT32 filesystem, we can't actually run it directly from the Terminal. Normally, you can. You'll need to run it through Extend Utils or do /bin/bash /my/mount/script.preMount instead.
Choose the script you saved in the previous part.
Type your sudo password.
That should be it.

For checking purposes:
Launch a Terminal.
Type mount
Look for a line at the bottom reading something like "aufs on /home type aufs (rw, si=67199377)" The number may be different... I didn't look much into it.
Type cd /tmp/NAMEOFOVERLAY.overlay for our example; cd /tmp/homeMount.overlay
We'll put a random file here to ensure it works touch MyTestFile
ls should show it up, as should ls ~

Now, Launch Debian Dev Extend.
Ok past the notifictions.
Select your Extend file.
Type your sudo password if prompted ( you may be quick enough to still be in the sudo cache that it doesn't ask )
In the Debian Dev Extend terminal, type gcc to ensure that works.
Type cd to jump directly to your home folder, and ensure our test file is there ls which if it is, you're good to go.

And now I gotta run!
If there's any issues, just post and I'll try and help :)

--further edit--
I can't remember why I split the thing between extend and overlay... that's confusing.. I'll get rid of that next release. I think it was because of OS Extends and having compressed versions of them, but that was a bit advanced and I don't think anyone uses them.
If you do, shout out! or I'll be removing them! Extends ARE Overlays, just that my menu system allows you to generate compressed Extends but not compressed Overlays. If you use Compressed Extends for whatever reason, let me know :) See, your head's probably exploded already with me mention Extend and Overlays so often ;) they're the same thing.

Edited by Stuckie, 28 June 2011 - 08:30 AM.


#17 Esn

Esn

    (:\

  • GP32 Hardcore
  • PipPipPipPipPipPip
  • 2758 posts
  • Gender:Male
  • Location:Toronto, Canada
  • Interests:Art, Classical Music, Biology, Fantasy/Sci-Fi (books, art, movies, games), Crosscountry skiing, adventuring to foreign parts, and of course, handheld gaming machines. ;)

Posted 28 June 2011 - 08:27 AM

Here's what shows up if I type in "mount" in the terminal spawned by the Debian Dev Extend pnd:

Spoiler


Also, in case it's of any use, here's what shows up if I type in "mount" in a normal terminal window launched outside the Debian extend:
Spoiler

Edited by Esn, 28 June 2011 - 08:33 AM.


#18 Stuckie

Stuckie

    GP Mania

  • GP32 Hardcore
  • PipPipPipPipPip
  • 432 posts
  • Gender:Male
  • Location:Tired, and sleeping in the basement...

Posted 28 June 2011 - 08:33 AM

Here's what shows up if I type in "mount" in the terminal spawned by the Debian Dev Extend pnd:


Oops, there's a bug... anyway, don't use "mount" in the Debian Dev Extend as it doesn't know what your base system is currently mounted.
That spam is from my testing it with the Old Extend Utils, which erroneously copied my mtab in, so it thinks your mount stuff is what I had mounted at the time ;)

Try it from an Angstrom terminal.
Also, I just posted a guide above you in my edited post, so try that too :)

#19 Esn

Esn

    (:\

  • GP32 Hardcore
  • PipPipPipPipPipPip
  • 2758 posts
  • Gender:Male
  • Location:Toronto, Canada
  • Interests:Art, Classical Music, Biology, Fantasy/Sci-Fi (books, art, movies, games), Crosscountry skiing, adventuring to foreign parts, and of course, handheld gaming machines. ;)

Posted 28 June 2011 - 08:35 AM

Ok, I edited my previous post to add in what appears if I type in "mount" from an Angstrom terminal. Thanks for the tutorial, I'll try to do things correctly this time. :)

Edited by Esn, 28 June 2011 - 08:37 AM.


#20 Stuckie

Stuckie

    GP Mania

  • GP32 Hardcore
  • PipPipPipPipPip
  • 432 posts
  • Gender:Male
  • Location:Tired, and sleeping in the basement...

Posted 28 June 2011 - 08:51 AM

You probably weren't far from it.. it's just a tad fiddly still.
I'm trying to figure out how people use the system ( whoever still does! ) so I can code up "wizards" to help run through it.

As well as tidy it up and get rid of confusing stuff like Extends vs Overlays, and preMount instead of extendMount.
I disappeared into work madness not long after releasing this version, so lots of niggles went uncaught. That and I had left the older version up, so that tended to be used instead, and I didn't really find out about most of these oddities. I shall be fixing all this now.. streamlining it and trying to bulletproof some of it so that maybe, just maybe, it might end up on the Repo as something useful ;)

And now I'm late.. oops.. oh well, hope that works for you!
Please post one way or another if it does or not :)

#21 Esn

Esn

    (:\

  • GP32 Hardcore
  • PipPipPipPipPipPip
  • 2758 posts
  • Gender:Male
  • Location:Toronto, Canada
  • Interests:Art, Classical Music, Biology, Fantasy/Sci-Fi (books, art, movies, games), Crosscountry skiing, adventuring to foreign parts, and of course, handheld gaming machines. ;)

Posted 28 June 2011 - 09:22 AM

Select "Mount Existing Pre-Mount"
Gotcha: As we've saved the script to a FAT32 filesystem, we can't actually run it directly from the Terminal. Normally, you can. You'll need to run it through Extend Utils or do /bin/bash /my/mount/script.preMount instead.
Choose the script you saved in the previous part.
Type your sudo password.
That should be it.

Please define "normally". Is that when the SD card is formatted to ext2/3/4? What about NTFS?

For checking purposes:
Launch a Terminal.
Type mount
Look for a line at the bottom reading something like "aufs on /home type aufs (rw, si=67199377)" The number may be different... I didn't look much into it.

Ok, the relevant line when I type that reads "aufs on /home type aufs (rw, si=67d14a37)". EDIT: The second time I tried it, the number was "61d6a177". I guess it constantly changes?

Type cd /tmp/NAMEOFOVERLAY.overlay for our example; cd /tmp/homeMount.overlay
We'll put a random file here to ensure it works touch MyTestFile
ls should show it up, as should ls ~

By "show it up", what do you mean? Here's my test of that:
esn-openpandora:/tmp/1gb.overlay$ touch uqm-0.7.0-content.uqm ls
esn-openpandora:/tmp/1gb.overlay$
Basically, nothing happens. Is that okay or did I do it wrong?

Oops, never mind, that was really silly of me. You meant use the "touch" command to create a new file called "mytestfile", and the "ls" command to verify that it exists. And I understood it differently. Yes, this step works as it should.

If you do, shout out! or I'll be removing them! Extends ARE Overlays, just that my menu system allows you to generate compressed Extends but not compressed Overlays. If you use Compressed Extends for whatever reason, let me know :) See, your head's probably exploded already with me mention Extend and Overlays so often ;) they're the same thing.

Well, I don't really mind as long as it's well-explained. THAT explanation though is a bit contradictory (extends=overlays, yet extends have different features, so therefore extends≠overlays? Uh? EDIT: Oh, nevermind... I guess you mean that it's just a different file extension so that the program recognizes which ones to compress and which ones not to). What would the potential use of a Compressed Extend be?
I do agree that "preMount" instead of "extendMount" is a bit confusing. When reading the wiki article, I was wishing for a section that just explained the unique terminology.

EDIT: I opened the Debian Extend after and tried to see what was in /home. No luck - still the same. Maybe I should just make a video of me doing it from start to finish? I'm following the instructions as far as I can tell, but I can't get the homeExtend to be the /home folder in the Debian Dev Extend. Will try it a few more times.

EDIT2: Okay, I tried it a few more times. Basically, everything works perfectly right up until the point when I type "ls" (or "dir") in the Debian Extend's home directory, and the contents it shows me are those of the /home directory of my NAND, not of the 1gb.overlay file.

Edited by Esn, 28 June 2011 - 10:10 AM.


#22 Stuckie

Stuckie

    GP Mania

  • GP32 Hardcore
  • PipPipPipPipPip
  • 432 posts
  • Gender:Male
  • Location:Tired, and sleeping in the basement...

Posted 28 June 2011 - 10:08 AM

Select "Mount Existing Pre-Mount"
Gotcha: As we've saved the script to a FAT32 filesystem, we can't actually run it directly from the Terminal. Normally, you can. You'll need to run it through Extend Utils or do /bin/bash /my/mount/script.preMount instead.
Choose the script you saved in the previous part.
Type your sudo password.
That should be it.

Please define "normally". Is that when the SD card is formatted to ext2/3/4? What about NTFS?

Normal to a Linux system.. I keep forgetting that in general, people with Pandoras will be using SD cards formatted for non-Linux systems.. so in this case, a Linux filesystem, not FAT or NTFS.

For checking purposes:
Launch a Terminal.
Type mount
Look for a line at the bottom reading something like "aufs on /home type aufs (rw, si=67199377)" The number may be different... I didn't look much into it.

Ok, the relevant line when I type that reads "aufs on /home type aufs (rw, si=67d14a37)". EDIT: The second time I tried it, the number was "61d6a177". I guess it constantly changes?

Type cd /tmp/NAMEOFOVERLAY.overlay for our example; cd /tmp/homeMount.overlay
We'll put a random file here to ensure it works touch MyTestFile
ls should show it up, as should ls ~

By "show it up", what do you mean? Here's my test of that:
esn-openpandora:/tmp/1gb.overlay$ touch uqm-0.7.0-content.uqm ls
esn-openpandora:/tmp/1gb.overlay$
Basically, nothing happens. Is that okay or did I do it wrong?

Oops, never mind, that was really silly of me. You meant use the "touch" command to create a new file called "mytestfile", and the "ls" command to verify that it exists. And I complete understood it differently, for some reason.

Did it work though?

If you do, shout out! or I'll be removing them! Extends ARE Overlays, just that my menu system allows you to generate compressed Extends but not compressed Overlays. If you use Compressed Extends for whatever reason, let me know :) See, your head's probably exploded already with me mention Extend and Overlays so often ;) they're the same thing.

Well, I don't really mind as long as it's well-explained. THAT explanation though is a bit contradictory (extends=overlays, yet extends have different features, so therefore extends≠overlays? Uh? EDIT: Oh, nevermind... I guess you mean that it's just a different file extension so that the program recognizes which ones to compress and which ones not to). What would the potential use of a Compressed Extend be?
I do agree that "preMount" instead of "extendMount" is a bit confusing. When reading the wiki article, I was wishing for a section that just explained the unique terminology.

Yep, going to fix that.. they're the same thing with a different extension is exactly what I meant.
The purpose of compressed extends was to provide something like a read-only compressed "base" and read-write "add-on" modules. The example in the original thread was something like a basic bootstrap of Debian as a compressed "base" and then things like a development toolchain in an "add-on" module, or GTK support or KDE, etc...

I should mention that whenever I restart my Pandora now (ever since first running the Debian Dev Extend), I get a terminal window called "Debian Dev Extend" popping up at startup with the following text:

We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others
    #2) Think before you type
    #3) With great power comes great responsibility.

Password:
EDIT: Sorry, nevermind, I was wrong - it happened a few times but this latest time it did not.

That's rather worrying.. how did you manage that? I don't save _anything_ to NAND to cause this...

EDIT2: Also, I opened the Debian Extend after and tried to see what was in /home. No luck - still the same. Maybe I should just make a video of me doing it from start to finish? I'm following the instructions as far as I can tell, but I can't get the homeExtend to be the /home folder in the Debian Dev Extend.


Ok.. this I think is where wires are getting crossed and confusion is reigning.
If you do ls ~ in your Angstrom terminal, do you see the file you created?
If you do ls ~ in your Debian terminal, do you see the file you created?

The Debian Extend mounts your personal home folder, not /home directly.
Your /home in Debian should have two folders - debian and esn ( I assume this is your user name! )
What I have been meaning by binding your home folder is in the Linux sense - your personal home folder, so /home/esn and not in the literal sense as in /home
Does that make better sense?

#23 Esn

Esn

    (:\

  • GP32 Hardcore
  • PipPipPipPipPipPip
  • 2758 posts
  • Gender:Male
  • Location:Toronto, Canada
  • Interests:Art, Classical Music, Biology, Fantasy/Sci-Fi (books, art, movies, games), Crosscountry skiing, adventuring to foreign parts, and of course, handheld gaming machines. ;)

Posted 28 June 2011 - 10:26 AM

timestamp='1309255730' post='952473' timestamp='1309255730' post='952473']


I should mention that whenever I restart my Pandora now (ever since first running the Debian Dev Extend), I get a terminal window called "Debian Dev Extend" popping up at startup with the following text:

We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others
    #2) Think before you type
    #3) With great power comes great responsibility.

Password:
EDIT: Sorry, nevermind, I was wrong - it happened a few times but this latest time it did not.

That's rather worrying.. how did you manage that? I don't save _anything_ to NAND to cause this...

It might've been because I restarted while having the terminal window open and "save session for future logins" ticked. It doesn't seem like any extends were actually mounted, though.

Ok.. this I think is where wires are getting crossed and confusion is reigning.
If you do ls ~ in your Angstrom terminal, do you see the file you created?

Yes

If you do ls ~ in your Debian terminal, do you see the file you created?

No.

Here's what the /tmp folder looks like in Xfe:

Posted Image

The /tmp/debextend/home/esn folder contains what is on my NAND, not what's in the 1GB home extend. There's a .pndXmodmap file in there, for example.

I'm working under the assumption that development work should be done within the Debian terminal, not within the Angstrom terminal. But how can I do this, if I can't access any of the files I want to work on from the Debian terminal?

You can use your Home Extend without issue.. bind the Home Extend first ( and if you're still using the older Extend Utils, use "mount" to make sure it actually _is_ bound as I had issues when I was testing with it and it didn't always take, ) then run the Debian Extend so that when it binds to your home folder, it binds to the right one ;) Else it can be sneaky and bind to NAND instead as I found out!

Yep, that's what's been happening. It has been binding to the NAND. Maybe FAT32 is causing the problem? Maybe I should I try putting everything onto an SD card formatted to ext2...

Edited by Esn, 28 June 2011 - 11:11 AM.


#24 Stuckie

Stuckie

    GP Mania

  • GP32 Hardcore
  • PipPipPipPipPip
  • 432 posts
  • Gender:Male
  • Location:Tired, and sleeping in the basement...

Posted 28 June 2011 - 12:26 PM

I've been using a FAT32 card to test this all with, so that's not the problem.

To be honest, I have no idea what the problem is :(
Especially as it looks like your home folder is bound out as /home/esn in the screenshot there, and since it's an rbind mount, it should be exactly what you see in Angstrom.

And I forgot to grab the Pandora on my way out, so I can't test anything till I get back.

#25 Esn

Esn

    (:\

  • GP32 Hardcore
  • PipPipPipPipPipPip
  • 2758 posts
  • Gender:Male
  • Location:Toronto, Canada
  • Interests:Art, Classical Music, Biology, Fantasy/Sci-Fi (books, art, movies, games), Crosscountry skiing, adventuring to foreign parts, and of course, handheld gaming machines. ;)

Posted 28 June 2011 - 12:57 PM

In case it helps figure out what's going on, here are a few more things.

Screenshot of the contents of /tmp/debextend/home/esn in Xfe:
Posted Image

pndrun_extendutils.out:
Spoiler

(I accidentally started it twice; the second time it recognized that the overlay was already mounted, and quit)

pndrun_DebianDevExtend.out:
Spoiler


Contents of 1gb.premount:
mkdir /tmp/1gb.overlay
gksudo "mount -o loop /mnt/utmp/extendutils/1gb.overlay /tmp/1gb.overlay"
gksudo "mount -t aufs -o dirs=/tmp/1gb.overlay=rw:/home=ro aufs /home"

Edited by Esn, 28 June 2011 - 12:59 PM.


#26 Stuckie

Stuckie

    GP Mania

  • GP32 Hardcore
  • PipPipPipPipPip
  • 432 posts
  • Gender:Male
  • Location:Tired, and sleeping in the basement...

Posted 28 June 2011 - 01:04 PM

Well there's another bug, extendutils isn't unmounting cleanly.. but that won't cause your issue.

From what I can see, everything is working as it should do :\
Let me see what the output of "mount" is from Angstrom, when everything is bound up.

#27 Esn

Esn

    (:\

  • GP32 Hardcore
  • PipPipPipPipPipPip
  • 2758 posts
  • Gender:Male
  • Location:Toronto, Canada
  • Interests:Art, Classical Music, Biology, Fantasy/Sci-Fi (books, art, movies, games), Crosscountry skiing, adventuring to foreign parts, and of course, handheld gaming machines. ;)

Posted 28 June 2011 - 01:09 PM

Sure. Here it is from the Angstrom terminal:

Spoiler


Here it is from the Debian terminal:

Spoiler


#28 Stuckie

Stuckie

    GP Mania

  • GP32 Hardcore
  • PipPipPipPipPip
  • 432 posts
  • Gender:Male
  • Location:Tired, and sleeping in the basement...

Posted 28 June 2011 - 01:21 PM

Yea ignore Debian's mount.. it's bollocks ;) it's my fault, and I might fix it, but probably not as it doesn't affect anything.

From your main Angstrom mount spew though, it does seem to have linked up successfully:
/dev/loop2 on /var/volatile/tmp/1gb.overlay type ext2 (rw,errors=continue)
aufs on /home type aufs (rw,si=6dcfdf37)
/dev/loop3 on /mnt/utmp/DebianDevExtend type iso9660 (ro)
none on /mnt/utmp/DebianDevExtend type aufs (rw,si=6dd67077,noplink)
/dev/loop4 on /var/volatile/tmp/debextend type ext2 (rw,errors=continue)
proc on /var/volatile/tmp/debextend/proc type proc (rw)
aufs on /var/volatile/tmp/debextend/home/esn type aufs (rw,si=6dcfdf37)

loop2 is mounting the overlay.
this is then aufs mounted to home.

loop3 is the Debian Extend PND.

loop4 is the Debian Extend itself
proc is bound
and /home/esn is bound from Angstrom into /home/esn on Debian.

The system says it's there and it's mounted.
I'm a bit at a loss as to what's gone wrong, now....

#29 Esn

Esn

    (:\

  • GP32 Hardcore
  • PipPipPipPipPipPip
  • 2758 posts
  • Gender:Male
  • Location:Toronto, Canada
  • Interests:Art, Classical Music, Biology, Fantasy/Sci-Fi (books, art, movies, games), Crosscountry skiing, adventuring to foreign parts, and of course, handheld gaming machines. ;)

Posted 28 June 2011 - 01:30 PM

and /home/esn is bound from Angstrom into /home/esn on Debian.

The Xfe screenshot shows that Debian's /home/esn is equivalent to the one that is normally on the NAND. Is it SUPPOSED to bind the home/esn folder that's normally on my NAND into the /home/esn folder on Debian? (If that's so, how exactly do I access the files in the 1gb extend? Where is it located, within the Debian extend?)

Edited by Esn, 28 June 2011 - 01:46 PM.


#30 Stuckie

Stuckie

    GP Mania

  • GP32 Hardcore
  • PipPipPipPipPip
  • 432 posts
  • Gender:Male
  • Location:Tired, and sleeping in the basement...

Posted 28 June 2011 - 01:47 PM

It does, but it does it via an aufs bind so you get your NAND /home as well as the overlay file.
All writes into /home will then go into the overlay rather than your NAND, but you can access them both from the same point.. which is one of the funky benefits of using overlays in general.

Therefore you should see everything that you'd normally get on your NAND, but you should _ALSO_ see what you've put in the overlay.
But you've said that you can't find the file you created on the overlay :\ so I don't know what it's up to as all the logs you've put up say that everything is working as it should be.