How to store a big set of data into an array in C

Go To StackoverFlow.com

0

I have a data file named somedata.dat, it contains a list of number pairs. about 10000 pairs. Like this:

3 19
5 213
1 34
7 93

I try to open the file, and read them, then put the numbers in a 10000x2 array. However, with the following code, it stops running when the loop hits about the 80000th time (needless to say EOF). Any ideas? Thanks.

int main(int argc, char *argv[])
{

int data[10000][2];
FILE *fp;

char s[5];
char temp[2];
char cur;
char next;
int pid=0;
int k=0;

fp = fopen("c:\\somedata.dat","r");
while (!EOF)
{
    cur = fgetc(fp);
    if (cur==' ')
    {           
        data[pid][0]=atoi(s);
        memset(&s[0], 0, sizeof(s));
    }
    else if (cur=='\n')
    {
        data[pid][1]=atoi(s);
        pid++;
        memset(&s[0], 0, sizeof(s));
    }
    else
    {
        temp[0]=cur;
        temp[1]='\0';
        strcat(s,temp);
    }
}
2012-04-04 07:27
by eastboundr
while (k!=80000) -- are you certain you didn't ask for it to stop working - sarnold 2012-04-04 07:29
im in no way C expert. but such big arrays should be in the heap. malloc it - Nahum 2012-04-04 07:29
You should use heap allocated arrays (with calloc) to deal with large data sets - Basile Starynkevitch 2012-04-04 07:29
You are not handling the return value of fgetc properly - Kerrek SB 2012-04-04 07:29
You should indent your code to better find matching brackets - Tarion 2012-04-04 07:30
sorry correction, it should be (!EOF) - eastboundr 2012-04-04 07:31
Hi Nahum, How do I malloc it? could you show some sample code? sorry i'm not good at - eastboundr 2012-04-04 07:33
And you read single chars but want to read multiple chars at once like "213". Have you tested your code with a smaller file - Tarion 2012-04-04 07:43
10000*2 = 20000 * sizeof(int), let us suppose 8, gives 160k, which is a ridiculous size (though, it may be still better to store it in the heap - ShinTakezou 2012-04-04 07:47
Yes smaller file works, it's just once the loop runs over about 80000 times, it stuck - eastboundr 2012-04-04 07:47
Hi ShinTakezou, could you show me how to declare this as 2-dimentional array with a sample code? thanks - eastboundr 2012-04-04 07:49


0

The array can be established and stored. However using malloc does make more sense. Thanks all.

2012-04-10 16:14
by eastboundr


1

Edit:

You define an array of 20,000 and try to add aprox 10,000 paris, and it stops at 80,000?

Would guess it is, since you have no break in the loop, it reads 10,000, then read EOF 70,000 times + do some strange stuff to the array. Then exits.


You have to compare EOF to something.

Typically:

int c; /* Has to be int, not char */

while((c = fgetc(fh)) != EOF) {

   ...
}

Also; take a look at fscanf, from stdio.h. Perhaps better suited for your usage.


As well; fgetc() retrieve one and one int representable as char or EOF.

I.e. File:

12 33
16 693

Then fgetc would retrieve:

1: '1'
2: '2'
3: ' '
4: '3'
5: '3'
6: '\n'
7: '1'
8: '6'
...

Further: Check when you use functions. If fgetc() return EOF, then you are either at EOF or a read error occurred. Check with feror etc.


2012-04-04 07:40
by Morpfh
Thanks, But do you know how to optimize the code so it does not stuck once the loop goes over 80000 times? thanks - eastboundr 2012-04-04 07:44
I would say, he must use fscanf in his case - ShinTakezou 2012-04-04 07:49
there's no intrinsic "80000" limits in that code. It will stop when the file will end (beware of buffer overflow - ShinTakezou 2012-04-04 07:50
@eastboundr If you have a file of 10,000 lines, and an array of 20,000 - You do not want to keep adding/reading after EOF. Or have I missed something here - Morpfh 2012-04-04 08:38
@user120115: use feof(fh) as a terminating condition of the loop - Basile Starynkevitch 2012-04-04 08:48


0

You have written:

while (k!=80000)

Might be the reason to stop at 80.000

2012-04-04 07:28
by Tarion
sorry should be (!EOF - eastboundr 2012-04-04 07:30
EOF is a constant, what is it's value? You should compare EOF with the data read - Tarion 2012-04-04 07:33
well, I just try to make the while loop reach the end of file... I thought !EOF does it.. - eastboundr 2012-04-04 07:35
When better use while(true) to avoid confusion. (or while(1) - Tarion 2012-04-04 07:40
@eastboundr EOF isn't a function, but a constant. If EOF is i.e. -1, then while(!EOF) would mean while (!-1) , or always false - Morpfh 2012-04-04 09:25
Ads