☰

거대한 순서의 텍스트 파일에서 두 문자열 사이의 텍스트 추출

tag-icon • tag-icon text-processing grep sort
거대한 순서의 텍스트 파일에서 두 문자열 사이의 텍스트 추출

우리는 수백만 개의 순서화된 타임스탬프 관측치를 포함하는 거대한 텍스트 파일을 가지고 있으며 시작점과 끝점이 주어지면 다음이 필요합니다.빠른 방법해당 기간에 대한 관측치를 추출합니다.

예를 들어 다음은 파일의 일부일 수 있습니다.

"2018-04-05 12:53:00",28,13.6,7.961,1746,104.7878,102.2,9.78,29.1,0,2.432,76.12,955,38.25,249.9,362.4,281.1,0.04
"2018-04-05 12:54:00",29,13.59,7.915,1738,104.2898,102.2,10.01,29.53,0,1.45,200.3,952,40.63,249.3,361.4,281.1,0.043
"2018-04-05 12:55:00",30,13.59,7.907,1734,104.0326,102.2,10.33,28.79,0,2.457,164.1,948,41.39,249.8,361.3,281.1,0.044
"2018-04-05 12:56:00",31,13.59,7.937,1718,103.0523,102.2,10.72,31.42,0,1.545,8.22,941,42.06,249.4,361.1,281.1,0.045
"2018-04-05 12:57:00",32,13.59,7.975,1719,103.1556,102.2,10.68,29.26,0,2.541,0.018,940,41.95,249.1,360.1,281.1,0.045
"2018-04-05 12:58:00",33,13.59,8,1724,103.4344,102.2,10.35,29.58,0,1.908,329.8,942,42.65,249.5,361.4,281.1,0.045
"2018-04-05 12:59:00",34,13.59,8,1733,103.9831,102.2,10.23,30.17,0,2.59,333.1,948,42.21,250.2,362,281.2,0.045
"2018-04-05 13:00:00",35,13.59,7.98,1753,105.1546,102.2,10.17,29.06,0,3.306,332.4,960,42,250.4,362.7,281.1,0.044
"2018-04-05 13:01:00",36,13.59,7.964,1757,105.3951,102.2,10.24,30.75,0,2.452,0.012,962,42.03,250.4,362.4,281.1,0.044
"2018-04-05 13:02:00",37,13.59,7.953,1757,105.4047,102.2,10.31,31.66,0,3.907,2.997,961,41.1,250.6,362.4,281.1,0.043
"2018-04-05 13:03:00",38,13.59,7.923,1758,105.4588,102.2,10.28,29.64,0,4.336,50.19,962,40.85,250.3,362.6,281.1,0.042
"2018-04-05 13:04:00",39,13.59,7.893,1757,105.449,102.1,10.27,30.42,0,1.771,12.98,962,41.73,249.8,362.1,281.1,0.043
"2018-04-05 13:05:00",40,13.6,7.89,1757,105.4433,102.1,10.46,29.54,0,2.296,93.7,962,43.02,249.9,361.7,281,0.045
"2018-04-05 13:06:00",41,13.59,7.915,1756,105.3322,102.1,10.52,29.53,0,0.632,190.8,961,43.64,249.3,361.5,281,0.045
"2018-04-05 13:07:00",42,13.6,7.972,1758,105.4697,102.1,10.77,29.49,0,0.376,322.5,961,44.69,249.1,360.9,281.1,0.046
"2018-04-05 13:08:00",43,13.6,8.05,1754,105.233,102.1,11.26,28.66,0,0.493,216.8,959,44.8,248.4,360.1,281.2,0.047

"2018-04-05 13:00:00"과 "2018-04-05 13:05:00" 사이의 데이터 포인트를 원하는 경우 출력은 다음과 같아야 합니다.

"2018-04-05 13:00:00",35,13.59,7.98,1753,105.1546,102.2,10.17,29.06,0,3.306,332.4,960,42,250.4,362.7,281.1,0.044
"2018-04-05 13:01:00",36,13.59,7.964,1757,105.3951,102.2,10.24,30.75,0,2.452,0.012,962,42.03,250.4,362.4,281.1,0.044
"2018-04-05 13:02:00",37,13.59,7.953,1757,105.4047,102.2,10.31,31.66,0,3.907,2.997,961,41.1,250.6,362.4,281.1,0.043
"2018-04-05 13:03:00",38,13.59,7.923,1758,105.4588,102.2,10.28,29.64,0,4.336,50.19,962,40.85,250.3,362.6,281.1,0.042
"2018-04-05 13:04:00",39,13.59,7.893,1757,105.449,102.1,10.27,30.42,0,1.771,12.98,962,41.73,249.8,362.1,281.1,0.043
"2018-04-05 13:05:00",40,13.6,7.89,1757,105.4433,102.1,10.46,29.54,0,2.296,93.7,962,43.02,249.9,361.7,281,0.045

grep또는 sed와 같은 기존 도구는 awk정렬된 파일에 적용하는 데 최적화되어 있지 않습니다. 그래서 그들은 충분히 빠르지 않습니다. 이진 검색을 사용하는 도구는 이러한 유형의 문제를 해결하는 데 적합합니다.

답변1

매우 큰 파일의 경우 유틸리티를 사용하면 look접두사 타임스탬프의 자연 순서를 활용하여 a 및 문자열의 가장 큰 공통 접두사에 대해 빠른 이진 검색을 수행할 수 있습니다. 그런 다음 이 작업을 수행/후처리하여 출력에서 ​​관심 라인을 추출할 수 있습니다.startendawksedlook

존재하다bash

export start='"2018-04-05 13:00:00"'
export end='"2018-04-05 13:05:00"'
#determine common prefix ("2018-04-05 13:0 in this example)
common_prefix=$(awk 'BEGIN {
   start=ENVIRON["start"]; end=ENVIRON["end"];
   len=length(start) > length(end)? length(end): length(start); 
   i=1;
   while (i <= len && substr(ENVIRON["start"], i, 1) == substr(ENVIRON["end"], i, 1)) {
       ++i
   }
    print(substr(start, 1, i-1))
}' </dev/null
)
#the -b option to look forces binary search. 
#My version of look on Ubuntu needs this flag to be passed, 
#some other versions of look perform a binary search by default and do not support a -b.
look -b "$common_prefix" file | awk '$0 ~ "^"ENVIRON["start"],$0 ~ "^"ENVIRON["end"]'

답변2

"2018-04-05 13:00:00"과 "2018-04-05 13:05:00" 사이의 줄을 인쇄합니다.

sed -n '/2018-04-05 13:00:00/,/2018-04-05 13:05:00/p' file

또는

sed -n /"2018-04-05 13:00:00"/,/"2018-04-05 13:05:00"/p file

시작 날짜 "2018-04-05 13:00:00"을 grep하고 다음 5줄(= 5분)을 출력하여 -m1첫 번째 일치 후 검색을 중지합니다.

grep -m1 -A5 '2018-04-05 13:00:00' file

관련 정보