Handling CSV Files in Bash – PL Courses

CSV (Comma Separated Values) is a popular file format used to store tabular data. In Bash, there are several ways to handle CSV files, including parsing and manipulating their contents. In this tutorial, we will explore various techniques for working with CSV files in Bash.

Reading CSV Files

Before we can start manipulating CSV files, we need to be able to read their contents. The most common approach is to use the `while` loop in Bash to read each line of the file. We can then use IFS (Internal Field Separator) to split each line into individual fields. Here’s an example:

#!/bin/bash

FILE="data.csv"

while IFS=',' read -r field1 field2 field3
do
    echo "Field 1: $field1"
    echo "Field 2: $field2"
    echo "Field 3: $field3"
done < "$FILE"

In this example, we assume that the CSV file has three fields separated by commas. The values of each field are assigned to variables (`field1`, `field2`, `field3`) which we can use for further processing.

If your CSV file contains a different delimiter, you can change the value of `IFS` accordingly. For example, if it uses a semicolon as the delimiter:

IFS=';'

If your CSV file contains quoted fields to handle values with commas or special characters, you can use the `csvtool` command-line tool to parse the CSV file. Make sure it is installed on your system before using it. Here’s an example:

#!/bin/bash

FILE="data.csv"

csvtool namedcol field1,field2,field3 "$FILE"

The `csvtool` tool automatically handles quoted fields, making it easier to parse CSV files with complex data.

Modifying CSV Files

Now that we know how to read CSV files in Bash, let’s explore how to modify their contents. Suppose we want to add a new column to our CSV file. We can use the `awk` command-line tool, which is built-in to most Unix-like systems, to accomplish this task. Here’s an example:

#!/bin/bash

FILE="data.csv"

awk -F ',' '{print [openai_gpt model="gpt-3.5-turbo-16k" max_tokens="14000" temperature="1" prompt="You are a skilled Bash developer. Write a comprehensive tutorial article on the topic of 'Handling CSV Files in Bash'. Focus directly on the specific Bash topic. The article should include detailed explanation of concepts and code examples. Write in an accessible style for coders. Format the article for embedding in a WordPress post. Use only 
, 
, 
, , 
, , 
 tags. Exclude headings and other HTML tags. When formatting code blocks, use this format: 
insert code here
. Ensure to convert the predefined characters '' (greater than) to HTML entities inside the 
 blocks! Do not insert HTML paragraphs inside the 
 blocks. Use carriage returns and spaces for formatting instead! DO NOT use any words like 'Intruduction', 'Counclusion' and all their synonyms! Keywords: csv parsing, manipulation"] ",New Column"}' "$FILE"
In this example, we use `awk` to specify the delimiter `-F` as a comma. We then append the desired value to each line using `print [openai_gpt model="gpt-3.5-turbo-16k" max_tokens="14000" temperature="1" prompt="You are a skilled Bash developer. Write a comprehensive tutorial article on the topic of 'Handling CSV Files in Bash'. Focus directly on the specific Bash topic. The article should include detailed explanation of concepts and code examples. Write in an accessible style for coders. Format the article for embedding in a WordPress post. Use only 
, 
, 

, ,


, , 
 tags. Exclude headings and other HTML tags. When formatting code blocks, use this format: 
insert code here
. Ensure to convert the predefined characters '' (greater than) to HTML entities inside the 
 blocks! Do not insert HTML paragraphs inside the 
 blocks. Use carriage returns and spaces for formatting instead! DO NOT use any words like 'Intruduction', 'Counclusion' and all their synonyms! Keywords: csv parsing, manipulation"] ",New Column"`. The modified CSV file is returned as the output.

If you want to remove a column from a CSV file, you can use the `cut` command-line tool. Here's an example:

#!/bin/bash

FILE="data.csv"

cut -d ',' -f 1-2 "$FILE"
In this example, we use `cut` to specify the delimiter `-d` as a comma and the fields to keep `-f 1-2`. The specified fields are extracted from each line, and the remaining columns are discarded.
Creating CSV Files
Sometimes, it's necessary to create a new CSV file from scratch. Bash provides several ways to achieve this. Here's an example using a simple loop:
#!/bin/bash

FILE="new_data.csv"

echo "Field 1,Field 2,Field 3" > "$FILE"

for i in {1..10}
do
    echo "Value $i,$((i * 2)),$((i + 10))" >> "$FILE"
done
In this example, we use `echo` to write the header line to the CSV file. We then use a loop to generate the values for each row and append them to the file using `>>`. Adjust the loop according to your requirements.
If you need to create complex CSV files with quoted fields and specific delimiters, you can use the `csvformat` command-line tool from the `csvkit` package. Make sure it is installed on your system before using it. Here's an example:
#!/bin/bash

FILE="new_data.csv"

echo "Field 1,Field 2,Field 3" > "$FILE"

for i in {1..10}
do
    echo ""Value $i",$((i * 2)),$((i + 10))" >> "$FILE"
done

csvformat -D ';' "$FILE" > "output.csv"
In this example, we generate a CSV file with quoted fields using `echo`. We then use `csvformat` to change the delimiter from a comma to a semicolon, and redirect the output to a new file named "output.csv". Adjust the quoting and delimiter options according to your needs.
In Bash, handling CSV files involves reading their contents, manipulating them, and creating new CSV files. By understanding the concepts and using the available command-line tools, you can easily parse and modify CSV files in your Bash scripts.
Happy CSV handling in Bash!






Source: https://www.plcourses.com/handling-csv-files-in-bash/


You might also like this video