Visual Studio Code – Problems encountered in recovering three tables from a dump file

I’m trying to restore tables from a dump file. It’s illustrated by a footnote in the paper "VCCFinder-Finding Potential Vulnerabilities in Open-Source Projects to Assist Code Audits", that the dump file that the team created with pg_dump could be read with pg_restore. As it’s shown in paper footnote with red line to emphasize. That’s where I’ve started.

1. Use pg_restore command

By typing the command mentioned in your paper: VCCFinder: Finding Potential Vulnerabilities in Open-Source Projects to Assist Code Audits:

pg_restore -f vcc_base I:OneDrivePractiseProjectx_prjsm_firmware_scanm_firmware_scan.refvcc-databasevccfinder-database.dump

Windows CMD had returned an error message:

pg_restore: error: input file appears to be a text format dump. Please use psql.

As I had tried the operation in different version, including v14.4, v9.6, v9.4 and v9.3, the outcome is the same error message.

2.Use psql command

Then I turned to another direction: using psql. After typing command,

psql -v ON_ERROR_STOP=1 -U postgres < I:OneDrivePractiseProjectx_prjsm_firmware_scanm_firmware_scan.refvcc-databasevccfinder-database.dump

apart from postgreSQL 14.4 environment, the returned error message is:

psql: SCRAM authentication requires libpq version 10 or above

Under postgreSQL 14.4 environment, the returned message became:

SET
SET
SET
SET
SET
SET
ERROR:  schema "export" already exists

If I remove the -v ON_ERROR_STOP=1 option, and returned message would be like this:

SET
SET
SET
SET
SET
SET
ERROR:  schema "export" already exists
SET
SET
SET
ERROR:  type "public.hstore" does not exist
LINE 27:     patch_keywords public.hstore
                            ^
ERROR:  relation "cves" already exists
ERROR:  relation "repositories" already exists
ERROR:  relation "commits" does not exist
invalid command n
invalid command N
invalid command N
...

(Solved) I have tried to solve the unreadable code problem shown in above error messages by typing chcp 65001, chcp 437 and etc to change character set into UTF8 or American English in Windows CMD, but it’s not helpful. But after viewing the source code of the dump file in Visual Studio, it’s not difficult to infer that those error messages were caused by psql commands in the dump file.

After the error messages became understandable, I focused on one particular error message:

ERROR:  type "public.hstore" does not exist
LINE 27:     patch_keywords public.hstore

So I manually created a "hstore" type below the "pulic SCHEMA", after that error messages turned into these:

SET
SET
SET
SET
SET
SET
SET
ERROR:  schema "export" already exists
SET
SET
SET
ERROR:  relation "commits" already exists
ERROR:  relation "cves" already exists
ERROR:  relation "repositories" already exists
ERROR:  malformed record literal: ""do"=>"1", "if"=>"0", "asm"=>"41", "for"=>"5", "int"=>"13", "new"=>"0", "try"=>"0", "auto"=>"0", "bool"=>"0", "case"=>"0", "char"=>"1", "else"=>"0", "enum"=>"0", "free"=>"0", "goto"=>"0", "long"=>"15", "this"=>"0", "true"=>"0", "void"=>"49", "alloc"=>"0", "break"=>"0", "catch"=>"0", "class"=>"0", "const"=>"0", "false"=>"0", "float"=>"0", "short"=>"0", "throw"=>"0", "union"=>"0", "using"=>"0", "while"=>"1", "alloca"=>"0", "calloc"=>"0", "delete"=>"0", "double"=>"0", "extern"=>"4", "friend"=>"0", "inline"=>"18", "malloc"=>"0", "public"=>"0", "return"=>"4", "signed"=>"1", "sizeof"=>"0", "static"=>"32", "struct"=>"4", "switch"=>"0", "typeid"=>"0", "default"=>"0", "mutable"=>"0", "private"=>"0", "realloc"=>"0", "typedef"=>"0", "virtual"=>"0", "wchar_t"=>"0", "continue"=>"0", "explicit"=>"0", "operator"=>"0", "register"=>"0", "template"=>"0", "typename"=>"0", "unsigned"=>"23", "volatile"=>"23", "namespace"=>"0", "protected"=>"0", "const_cast"=>"0", "static_cast"=>"0", "dynamic_cast"=>"0", "reinterpret_cast"=>"0""
DETAIL:  Missing left parenthesis.
CONTEXT:  COPY commits, line 1, column patch_keywords: ""do"=>"1", "if"=>"0", "asm"=>"41", "for"=>"5", "int"=>"13", "new"=>"0", "try"=>"0", "auto"=>"0", "bo..."
ERROR:  syntax error at or near "l022_save"
LINE 1: l022_save, pl022_load, s);
        ^
invalid command n
invalid command N
invalid command N
...

Now the three tables have been created, but there is no content in them.

3. Install hstore

After searching for "hstore"hstore type does not exist with hstore installed postgresql, I realized that the "hstore" should be installed, but not be manually created. So I typed this in psql command line:
postgres=# create EXTENSION hstore; And there were new error messages:

SET
SET
SET
SET
SET
SET
SET
ERROR:  schema "export" already exists
SET
SET
SET
CREATE TABLE
ERROR:  relation "cves" already exists
ERROR:  relation "repositories" already exists
ERROR:  missing data for column "hunk_count"
CONTEXT:  COPY commits, line 23201: "11388700   178     N      other_commit    1d6198c3b01619151f3227c6461b3d53eeb711e5N      blueswir1@c046a42c-6fe2-441..."
ERROR:  syntax error at or near "l022_save"
LINE 1: l022_save, pl022_load, s);
        ^
invalid command n
invalid command N
invalid command N
...

And still, there is no content in those three tables.

4. Generate and view tables

After looking into the source code of the dump file, and trying to fix the "hunk_count" problem but end up with failure. It occurs to me that the above error messages just caused by one paticular row of code. So I had deleted the row and the old error messages were gone but there were new error messages caused by another row. Evetually I have deleted 10 rows in total, comparing to the total row number: 351409, those deleted parts are negligible. And three tables weren’t empty anymore, as it’s shown in pgAdmin 4.

However, the pgADmin only demonstrated the structure of those tables, I still didn’t know how to view the content in them. By refering to 2 Ways to View the Structure of a Table in PostgreSQL, I typed

SELECT 
   *
FROM 
   export.repositories/ export.cves/ export.commits
WHERE 
   TRUE

to generate and view corresponding tables in pgAdmin 4. For example, final cve table:

5. In the end

Looking back at these steps, these are all easy steps, but for a guy who was not familiar with the tools or operations, it could cost several days to search and type, step by step for one simple purpose. I wish this post could be useful to someone like me.

However, I am not so familiar with psql commands or anything about postgreSQL, as a matter of fact, I had never used them before. So I’m wondering if someone could point out some mistakes I may have made in those attempts, or provide some suggestions for my dilemma.