当前位置 主页 > 网站技术 > 代码类 >

    python 多进程并行编程 ProcessPoolExecutor的实现

    栏目:代码类 时间:2019-10-30 15:03

    使用 ProcessPoolExecutor

    from concurrent.futures import ProcessPoolExecutor, as_completed
    import random

    斐波那契数列

    当 n 大于 30 时抛出异常

    def fib(n):
      if n > 30:
        raise Exception('can not > 30, now %s' % n)
      if n <= 2:
        return 1
      return fib(n-1) + fib(n-2)

    准备数组

    nums = [random.randint(0, 33) for _ in range(0, 10)]
    '''
    [13, 17, 0, 22, 19, 33, 7, 12, 8, 16]
    '''
    

    方案一:submit

    submit 输出结果按照子进程执行结束的先后顺序,不可控

     with ProcessPoolExecutor(max_workers=3) as executor:
        futures = {executor.submit(fib, n):n for n in nums}
        for f in as_completed(futures):
          try:
            print('fib(%s) result is %s.' % (futures[f], f.result()))
          except Exception as e:
            print(e)
    '''
    fib(13) result is 233.
    fib(17) result is 1597.
    fib(0) result is 1.
    fib(22) result is 17711.
    fib(19) result is 4181.
    can not > 30, now 33
    fib(7) result is 13.
    fib(12) result is 144.
    fib(8) result is 21.
    fib(16) result is 987.
    
    '''

    等价写法:

     with ProcessPoolExecutor(max_workers=3) as executor:
        futures = {}
        for n in nums:
          job = executor.submit(fib, n)
          futures[job] = n
    
        for job in as_completed(futures):
          try:
            re = job.result()
            n = futures[job]
            print('fib(%s) result is %s.' % (n, re))
          except Exception as e:
            print(e)
    '''
    fib(13) result is 233.
    fib(17) result is 1597.
    fib(0) result is 1.
    fib(22) result is 17711.
    can not > 30, now 33
    fib(7) result is 13.
    fib(19) result is 4181.
    fib(8) result is 21.
    fib(12) result is 144.
    fib(16) result is 987.
    '''

    方案二:map

    map 输出结果按照输入数组的顺序

    缺点:某一子进程异常会导致整体中断

     with ProcessPoolExecutor(max_workers=3) as executor:
        try:
          results = executor.map(fib, nums)
          for num, result in zip(nums, results):
            print('fib(%s) result is %s.' % (num, result))
        except Exception as e:
          print(e)
    '''
    fib(13) result is 233.
    fib(17) result is 1597.
    fib(0) result is 1.
    fib(22) result is 17711.
    fib(19) result is 4181.
    can not > 30, now 33
    '''

    以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持IIS7站长之家。